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IJWA  EQKMS  Data  Classification  Schema 


1.0  Introduction 

The  discipline  of  Knowledge  Management  (KM)  is  new  but  emerging  rapidly  in  the  commercial, 
industrial  and  government  sectors  of  the  economy  where  there  is  a  need  for  insight  on  consumer- 
business,  business-business,  and  agency-agency  decisions.  The  situation  in  the  military  is  several 
layers  of  complexity  beyond  the  consumer  market  and  must  address  not  only  mission-critical 
data  but  do  so  within  the  scope  of  an  extremely  dynamic  environment.  Corporations  build 
knowledge  management  systems  to  understand  overall  market  trends  that  might  affect  the 
company  over  the  next  several  quarters.  Military  systems  are  far  more  complex,  managing 
knowledge  from  many  more  sources — with  data  streams  from  a  variety  of  technologies,  input 
from  divergent  perspectives,  and  very  short  time  frames  in  which  to  make  critical  decisions. 

To  support  the  needs  of  the  military,  IJWA  is  categorizing  information  within  a  knowledge 
repository.  IJWA  collects  both  qualitative  and  quantitative  data  in  their  support  of  FBEs.  The 
IJWA  Ethnographic  Qualitative  Knowledge  Management  System  (EQKMS)  is  a  repository  for 
reports,  interviews,  chats,  observations,  and  other  FBE  qualitative  information.  The  first  sections 
of  this  report  address  the  qualitative  knowledge  management  processes  under  development 
through  the  EQKMS  project  and  the  later  sections  discuss  the  relationship  between  qualitative 
and  quantitative  data  schemas,  integration  methodologies  for  FBE  data,  and  present  and  future 
capabilities. 


2.0  KM  for  Analysis  of  Operational  Capabilities 

The  term  “Knowledge  Management”  is  used  here  in  the  broadest  sense.  We  refer  to  managing 
numerical  values  obtained  from  an  automated  collection  system,  human  subjective  opinions, 
synthesis  results,  results  tailored  to  address  specific  long-range  initiatives,  etc.  The  challenge  is  to 
create  a  knowledge  management,  KM,  system  that  enables  archival,  retrieval,  and  analysis.  This 
paper  describes  a  KM  system  that  supports  the  analysis  of  military  capabilities. 

The  information  to  be  archived  in  this  system  come  from  differing  sources:  studies,  wargames, 
and  field  experiments.  A  characteristic  of  these  events  is  their  variability.  They  have  neither  a 
common  structure  nor  a  common  core  of  assumptions.  In  fact,  there  is  an  overt  desire  to  test  a 
range  of  operational  structures  and  situations  so  that  even  a  given  type  of  event,  such  as  Fleet 
Battle  Experiments,  will  have  some  of  its  conditions  change  from  event  to  event.  On  the  other 
hand,  there  is  a  desire  to  synthesize  results  from  many  events  to  obtain  conclusions  on  current 
and  future  operational  capabilities.  This  means  that  the  KM  system  must  be  robust  to  changes 
in  the  configurations  under  which  information  is  obtained  and  developed. 

The  most  important  step  in  creating  the  KM  system  is  to  insure  that  it  supports  the  major 
strategic,  operational,  and  tactical  questions  being  addressed,  or  that  may  later  be  addressed,  by 
the  events.  The  KM  system  described  here  has  been  created  specifically  for  Fleet  Battle  Experi¬ 
ments.  Fortunately,  Fleet  Battle  Experiments  are  very  broad  in  configuration  and  the  issues  they 
address  so  it  has  been  relatively  easy,  using  them  as  a  model,  to  develop  the  required  KM  system. 
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Within  DoD,  there  are  a  relatively  small  number  of  overarching  concepts  under  consideration. 
For  a  high-level  organizing  structure,  we  use  concepts  such  as: 

Joint  Maritime  Access  Control  (JMAC) 

Time  Critical  Targeting  (TCT) 

Theater  Ballistic  Missile  Defense  (TBMD) 

Full  Dimension  Protection  (FDP) 

Mine  Counter  Measures  (MCM) 

Network  Centric  Warfare  (NCW) 

Common  Operations  Picture  (COP) 

These  are  an  illustration,  not  an  all-inclusive  list.  They  are  important  operational  concepts  that 
support  multilevel  questions,  including  high-level  questions  such  as  “should  we  operate  in  the 
littoral?”,  “can  we  support  widely  dispersed  troops  ashore?”. 

Examining  the  above  list,  one  sees  immediately  that  the  concepts  are  not  independent  nor  are 
they  of  the  same  type.  JMAC  is  a  strategic  goal  and  TCT,  TBMD,  FDP,  and  MCM  are  opera¬ 
tions  needed  in  support  of  that  goal.  NCW  is  an  information  superiority  concept  which  can  aid 
or  enable  operations  rather  than  a  type  of  operation.  COP  is  a  needed  tool  within  NCW  Even 
though  there  are  structural  differences  between  these  “concepts”,  they  are  often  treated  as  being 
at  the  same  level  when  planning  a  complex  event.  This  is  not  a  problem  as  long  as  one  recog¬ 
nizes  the  differences  and  sets  up  a  KM  structure  which  allows  the  proper  relationships  between 
them  when  analyzing  the  events. 

Developing  a  KM  system  requires  that  there  be  an  archiving  methodology  which  supports  the 
“thread  pulling”  method  used  for  developing  and  retrieving  information.  When  archiving  we 
place  several  appropriate  “tags”  on  each  piece  of  data.  (We  are  using  the  term  “data”  very  broadly 
here).  Information  is  retrieved  by  “pulling”  on  a  set  of  tags,  which  we  refer  to  as  thread  analysis. 
Thread  analysis  starts  with  a  specific  question,  from  which  a  set  of  tags  is  defined  to  pull  the 
appropriate  data.  The  data  archive  must  be  as  robust  as  possible  with  respect  to  thread  analysis 
for  a  wide  range  of  questions,  i.e.,  the  system  must  be  set  up  so  that  one  can  access  every  piece  of 
data  that  has  applicability  to  a  given  area  of  inquiry.  This  requires  an  extensive  set  of  tags  and 
several  tags  on  each  datum.  If  the  tagging  system  is  not  set  up  wisely,  the  number  of  tags  needed 
on  a  particular  datum  can  get  out  of  hand.  We  have  found  that  the  answer  to  this  dilemma  of 
the  need  to  balance  robustness  and  cumbersome  tagging  is  to  take  an  object  oriented  approach, 
which  is  described  later  in  this  document. 

Knowledge,  information,  data,  regardless  of  the  semantics  used,  occur  in  a  hierarchy  or  at  levels. 
There  are  no  hard  and  fast  rules  for  the  number  of  levels  and  how  they  are  defined.  The  number 
depends  on  the  granularity  desired  for  information.  The  definition  depends  on  the  specifics  of 
the  system  being  examined.  In  general,  too  many  levels  (fine  grained)  leads  to  an  overly  complex 
KM  system  which  is  arduous  and  time  consuming  to  use. 

For  our  purposes,  we  prefer  three  levels.  They  are: 

Level-1  -  objective  and  subjective  data  that  directly  address  events  (event  data). 

Level-2  -  conclusions  concerning  the  performance  of  a  system  (system  information). 
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Level-3  -  conclusions  that  address  capabilities  at  the  initiative  level  (results). 


Note  that  we  are  now  discriminating  between  the  terms  data,  information,  and  results. 

Level- 1  data  consists  of  event  descriptions  and  the  time  at  which  events  occur.  Data 
can  be  obtained  from  an  automated  acquisition  system  or  from  an  observer  record¬ 
ing  an  occurrence.  Data  also  includes  observations  of  the  status  of  systems,  work¬ 
arounds,  configuration  changes,  etc.  that  occur  at  a  particular  time. 

Level-2  information  often  is  a  subjective  opinion,  but  it  can  also  be  a  conclusion 
developed  from  Level- 1  data.  There  is  no  time  associated  with  them  but  they  should 
be  in  the  “context”  of  a  particular  operation,  platform,  command  and  control 
configuration,  etc.  These  contexts  can  change  with  time.  Context  information  is 
often  referred  to  as  “meta-data”.  As  noted  above,  Level-2  data  refers  to  systems. 

“System”  is  not  meant  to  apply  only  to  hardware.  It  often  will  refer  to  a  C2  system, 
and  can  also  refer  to  a  process.  The  only  requirements  for  something  to  be  called  a 
system  are  that  it  be  an  identifiable  entity  and  that  the  interrelationships  between  its 
components  can  be  defined.  One  must  also  be  able  to  identify  the  interactions 
between  the  system  and  its  external  world. 

Level-3  results  will  most  often  be  pulled  from  Levels  1  and  2  through  thread  analy¬ 
sis.  Expert  opinion  may  also  be  directly  inputted  to  the  KM  system  database. 

When  this  is  done,  developing  supporting  information  from  Levels  1  and  2  should 
be  done  or  the  validity  of  the  results  may  be  suspect. 

It  is  important  to  recognize  that  questions  have  levels  in  order  to  couple  questions  to  the  proper 
thread-pulling  analysis  from  the  KM  system.  Question  levels  are  not  the  same  as  data  levels,  and 
the  answer  to  a  given  question  will  usually  require  accessing  more  than  one  KM  level.  Examples 
most  easily  illustrate  this.  Consider  two  questions: 

1 .  Does  a  particular  system  (or  method)  shorten  the  TCT  time  line. 

2.  Does  a  particular  COP  configuration  aid  in  reducing  the  TCT  time  line. 

The  first  question  is  straightforward,  and  the  answer  requires  pulling  the  appropriate  Level- 1 
data,  in  particular  the  times  required  to  perform  the  various  TCT  processes.  One  requirement  is 
that  there  be  a  performance  baseline  from  which  the  comparison  can  be  made  if  sense  is  to  be 
made  of  “shorten”.  The  second  question  is  more  complex.  Answering  it  can  require  that  one 
pull  data  and  information  from  Levels- 1  and  -2,  then  attempt  to  isolate  the  influence  of  the 
COP  from  other  factors.  The  answers  to  these  questions  can  provide  KM  system  information 
and  results  at  Levels-2  and  -3. 

2.1  Question,  Data-level,  And  System  Relationships 

The  purpose  of  the  KM  system  is  to  support  analysis  of  operational  capabilities  through  the 
examination  of  processes  and  systems.  Questions  form  the  basis  for  analysis  and  are  the  key  to 
reaching  into  the  KM  system.  Information  may  be  desired  about  a  system  that  provides  an  end- 
to-end  capability,  or  one  of  its  subsystems.  The  question  could  concern  the  effectiveness  of  task 
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performance,  perhaps  using  an  MOE  such  as  time,  or  it  could  be  the  value  of  a  particular  param¬ 
eter,  such  as  reliability.  One  must  devise  the  three-level  KM  system,  and  the  associated  data  tags, 
so  that  a  wide  diversity  of  questions  can  be  supported. 

The  relationships  between  and  within  KM  system  levels  are  important.  A  systematic  methodol¬ 
ogy  is  needed  to  aggregate  data  in  Level- 1  into  information  for  Level-2.  There  must  be  coher¬ 
ence  between  opinions  inserted  directly  into  Level-2  and  the  information  pulled  from  Level- 1. 
The  same  is  true  for  the  relationships  between  Level-2  information  and  Level-3  results. 

Tags  placed  on  information  and  data  are  the  keys  to  accessing  them.  The  analysis  methodology 
that  allows  one  to  go  from  a  question  to  the  correct  set  of  keys  to  obtain  the  desired  answer  must 
be  logical,  reasonably  transparent,  and  fairly  easy  to  use.  The  basic  requirements  for  a  viable  data 
tagging  scheme  are: 

•  an  easy  correspondence  between  questions  addressed  to  a  particular  KM  level  and  the  tags 

•  a  logical  relationship  between  the  tags  for  the  three  levels 

The  relationships  between  the  types  of  questions  and  the  three  KM  Levels  are  fairly  straightfor¬ 
ward  when  one  refers  to  the  definition  of  the  Levels  given  above.  Where  the  various  types  of 
system  information  can  be  found  and  how  that  relates  to  questions  is  more  complex.  To  illus¬ 
trate  we  consider  possible  questions  that  address  the  two  Levels.  First,  two  Level- 1  system 
questions: 

a.  How  long  does  it  take  from  detection  of  a  time  critical  target  to  the  time  when  weapon/target  pairing 

is  completed? 

b.  How  long  does  it  take  to  perform  target  mensuration? 

Both  are  Level- 1  questions  because  they  refer  to  the  value  of  a  particular  system  parameter,  time 
for  both  questions.  The  first  time  can  be  obtained  by  summing  the  processing  times  for  the 
appropriate  parts  of  the  total  system.  It  may  be  necessary  to  follow  sets  of  events  through  the 
system  to  obtain  the  times,  or  the  processing  times  may  be  archived.  The  second  time  is  ob¬ 
tained  by  pulling  data  for  the  subsystem  that  performs  mensuration. 

Second,  consider  two  Level-2  questions: 

a.  Does  incorporation  of  an  ISR  desk  to  manage  sensors  improve  the  TCT  process? 

b.  Does  an  ISR  desk  improve  the  quality  of  forwarded  target  folders? 

Both  are  Level-2  questions  because  they  do  not  ask  for  a  parameter  or  MOE,  but  for  information 
about  system  performance.  The  first  question  concerns  a  macro-process,  TCT,  and  the  second 
for  a  sub-process,  target  folders  generation. 

It  may  be  that  the  answers  to  the  Level-2  questions  can  be  obtained  by  reaching  into  only  infor¬ 
mation  in  Level-2  or  it  may  be  necessary  to  also  pull  out  some  Level- 1  data.  Note  that  both 
questions  contain  the  word  “improve”.  This  implies  that  a  comparison  is  needed  between 
performing  a  process  with  and  without  and  ISR  desk,  which  means  that  somewhere  there  must 
be  baseline,  or  without  the  desk,  information.  This  means  that  information,  or  tags,  must  be 
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present  in  the  KM  system  that  identifies  the  configurations  that  were  in  use  when  data/informa¬ 
tion  were  collected.  The  above  examples  illustrate  that  a  fair  amount  of  care  is  needed  to  relate 
questions  and  the  tags. 

The  fact  that  Level-2  data  will  contain  subjective  system  information  implies  that  the  informa¬ 
tion  will  not  make  sense  unless  one  has  a  good  definition  of  what  the  system  is.  This  is  true,  and 
especially  important  for  C2I  systems  because  they  are  in  a  near-continuous  state  of  evolution. 
Thus,  maps  of  the  various  C2I  systems  are  required  as  supporting  information  for  the  data 
archive  (accessed  through  tags  on  the  data  and  information).  In  addition,  there  are  many  hard¬ 
ware  and  software  differences  between  experiments,  so  that  supporting  configuration  informa¬ 
tion  and  tags  must  also  be  included. 

2.2  Data  Tagging  Structure 

The  tagging  structure  must  map  in  a  fairly  transparent  way  on  the  objects  and  events  that  make 
up  military  operations.  The  number  of  categories  should  be  small  to  reduce  complexity,  and 
there  should  be  no  overlapping  categories,  which  would  create  uncertainty  in  how  to  tag  and 
make  information  retrieval  difficult.  These  criteria  can  be  accomplished  by  defining  three 
categories: 

Things  -  objects,  systems,  or  people  that  perform  actions. 

Attributes  -  descriptions  of  the  state  or  characteristics  of  things. 

Actions  -  activities  that  occurs  at  a  particular  time  that  change  the  state  of  a  thing 
or  are  interactions  between  things. 

There  are  subtleties  involved  in  using  a  simplified  tagging  structure  of  this  type.  For  example, 
suppose  the  piece  of  information  to  be  archived  is  an  action  taken  by  an  object.  The  obvious 
tags  are  those  that  identify  the  object  and  the 
action  taken.  In  addition,  it  is  probably  important 
to  know  the  attributes  of  the  object  to  place  the 
information  in  the  proper  context.  Thus  tags  from 
all  three  categories  can  be  needed  for  the  datum. 

Almost  never  will  data  be  tagged  with  only  one  of 
the  above  categories. 

Within  these  three  categories  it  is  possible  to 
identify  the  sets  of  Things,  Attributes,  and  Actions 
that  will  adequately  describe  military  operations 
(Figure  1). 

Some  of  these  are  obvious,  some  not,  and  examina¬ 
tion  of  the  lists  could  lead  one  to  conclude  some  are  downright  strange.  A  full  description  of  the 
underlying  rationale  is  beyond  the  scope  of  this  paper,  but  a  few  examples  to  illustrate  the  basic 
ideas  are  warranted. 


Thinqs 

Attributes 

Actions 

Platform 

Status 

Transmit 

Sensor 

Mission 

Receive 

Weapon 

Location 

Detect 

Information 

Cmd  Relation 

Decide 

C2  System 

Assignment 

Command 

Force 

Fire 

Cmd  Center 

Reposition 

Figure  1 .  Level  1  objective  and  subjective  data  that 
directly  address  events. 
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C2  system  refers  to  the  full  system. 

Command  Node  is  an  activity  that  issues  commands,  be  it  a  single  person  or  CIC. 

Physical  refers  to  any  physical  action,  such  as  fire  or  reposition. 

Command  is  the  issuance  of  a  command  at  its  source. 

Transmit  refers  to  sending  any  information,  including  commands. 

A  force  is  any  size  grouping  of  platforms. 

Workload  refers  to  how  much  activity  a  thing  has  to  perform. 

Capacity  is  the  current  ability  to  carry  out  its  activity. 

Capacity  can  also  be  physical,  such  as  how  many  rounds  in  a  magazine. 

Obviously,  there  are  many  sub-tags  within  each  of  these  descriptors.  There  are  types  and  identifi¬ 
ers,  e.g.,  platform  includes  a  tag  for  the  type,  such  as  ship  or  airplane  and  the  identification  of  the 
specific  platform.  Decide  could  be  a  decision  to  engage  a  target  or  it  could  be  a  decision  made 
for  BDA  assessment.  Data  and  information  will  have  tags  that  identify  where  they  fit  within 
these  categories.  A  given  piece  of  data  will  have  more  than  one  tag,  e.g.  to  tag  sensor  information 
being  sent:  sensor,  platform,  sensor  type,  location,  sensor  information,  transmission. 

2.3  System  Definition 

The  purpose  of  this  section  is  not  to  define  the  word  system,  but  to  indicate  how  one  links  a 
question  or  thread  analysis  to  what  one  considers  to  be  the  system  for  the  particular  case.  In 
much  of  what  follows  in  this  paper  we  use  “sensor  system”  as  the  example,  broadly  defined  to  be 
all  components  and  actions  from  the  point  at  which  a  target  is  detected  to  the  point  at  which 
weapon/target  pairing  is  accomplished.  With  this  definition,  one  can  list  those  functions  per¬ 
formed  by  this  system: 


Sensor  System 

Functions  (Actions) 

At  Sensor  Platform 

At  ISR  Center 

Receive  Command 

Receive  Data 

Move  Platform 

Fuse  Data 

Command  Sensor 

Interpret  Data  /  Decide 

Search 

Assign  Sensor 

Detect 

Create  Target  Folder 

Transmit  Data 

Send  Folder 

Mensurate 

Nominate  Target,  Transmit 

The  above  assumes  that  there  is  some  type  of  central  function,  ISR  Center,  or  more  than  one, 
that  receives  sensor  information  and  acts  on  it.  However,  the  listed  functions  are  independent  of 
the  exact  structure  of  the  sensor  system.  The  functions  are  shown  in  the  order  that  data  is 
developed  by  the  system,  starting  with  the  sensor  on  a  platform  being  moved  into  position, 
through  detection  and  transmitting  information,  processing  the  information  at  an  ISR  site,  and 
sending  the  final  target  nomination  out  of  the  system. 

Having  defined  the  system,  it  is  fairly  easy  to  see  the  tags  that  would  be  attached  to  data  for  one 
of  these  functions.  There  would  be  significant  number  of  meta-tags  that  identify  the  experiment, 
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platform,  and  other  context.  Then  there  is  the  Sensor  tag  to  identify  the  data  as  belonging  to  the 
sensor  class,  followed  by  other  tags  to  designate  specifics,  such  as  it  refers  to  transmission  of  a 
command  to  the  sensor  platform  from  the  ISR  desk. 

A  datum  will  have  only  one  set  of  meta-tags,  but  it  can  have  more  than  one  set  of  system  tags. 

For  example,  information  probably  will  be  associated  with  more  than  one  system,  such  as  the 
sensor  system  and  the  COP.  Thus,  tags  appropriate  to  both  will  be  present.  This  allows  inter¬ 
system  relationships  to  be  examined.  An  example  could  be  how  a  specific  sensor  control  configu¬ 
ration  contributes  to  an  improved  COP. 

2.4  Level-3:  Results  and  Analysis 

Level-3  results  address  capabilities  at  the  operational  level.  Thus  the  tags  will  be  derived  from 
major  operational  concepts,  such  as: 

TCT  Time  Critical  Targets 

STOM  Ship  to  Objective  Maneuver 

NCW  Network  Centric  Warfare 

COP  Common  Operating  Picture 

CAS  Close  Air  Support 

These  results  will  have  been  obtained  from  a  single  experimentation  event  or  by  synthesis  of 
results  from  several  events.  In  either  case,  the  result  in  the  database  needs  to  have  a  tag  that 
identifies  the  event(s).  It  is  not  only  possible,  but  probable  that  information  that  applies  to  one 
concept  can  apply  to  one  or  more  others.  Thus  the  result  will  have  tags  for  each  of  the  concepts 
to  which  it  applies. 

Many  of  the  results  will  have  been  developed  from  information  in  Level-2  or  even  from  data  in 
Level- 1.  It  is  important  to  identify  the  trail(s)  through  the  data  from  which  the  result  was 
developed.  Tags  are  also  used  for  this  purpose.  This  allows  one  to  access  the  supporting  evidence 
for  a  result. 

The  best  way  to  understand  the  relationships  between  analysis,  the  data  system  structure,  and  the 
use  of  tags  is  to  consider  an  example  analysis  question.  The  following  is  a  constructed,  rudimen¬ 
tary  example  of  the  process,  presented  as  a  set  of  logical  steps.  It  illustrates  a  thread. 

Analysis  Question: 

How  well  can  we  do  TCT?  (Note  poorly  constructed,  broad  question) 

Results  Pull:  Pull  TCT  tagged  data  from  Level-3. 

Results:  A  is  good.  B  is  not  so  good. 

Concurrent  FDP  and  TCT  with  the  same  platform  is  difficult. 

These  pulled  results  may  be  sufficient  as-is  or  one  may  wish  to  use  them  as  a  starting  point  to 
explore  more  deeply.  Then,  one  needs  to  ask  a  more  in-depth,  specific  question  and  do  another 
information  pull,  probably  from  Level-2  and  even  Level- 1.  Continuing  with  this  example, 
assume  the  interest  is  in  the  concurrent  FDP/TCT  result. 
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Analysis  Question:  "What  interaction  between  TCT  and  FDP  reduces  the 

ability  to  do  TCT? 

Information  Pull:  Pull  TCT  and  FDP  tagged  information  from  Level-2. 

Only  pull  information  that  has  the  same  platform  tag. 
Information:  Difficulties  with  tube  loading.  Insufficient  SAM 

rounds.  TCT  C2  configuration  too  slow  for  FDP 

This  information  focuses  one  on  one  or  several  aspects  of  the  problem.  At  this  point  a  third  (or 
more)  analysis  questions  can  be  posed.  In  this  way  the  thread  of  information  is  built  up. 

3,d  Pulls:  Pull  the  connected  C2,  weapon  system,  sensor  system: 

information  from  Level-2,  and  data  from  Level- 1. 

Other  iterations  in  the  process  will  occur  until  the  analyst  is  satisfied  with  the  information  or 
there  is  nothing  new  to  be  found.  A  result  of  this  analysis  may  be  to  create  new  results  and 
information,  and  archive  them  with  the  appropriate  tags. 

The  above  example  focuses  on  using  the  database  for  analysis,  starting  with  results  that  are 
already  in  Level-3.  In  order  to  have  results  in  Level-3,  analyses  may  have  already  been  done.  It  is 
also  possible  that  the  results  were  inserted  directly  from  expert  observations  made  during  an 
experiment.  This  introduces  the  need  for  two  types  of  tags  for  Level-3  results.  If  the  result  has 
been  inserted,  the  tags  will  identify  the  concept  and  whatever  context  is  needed.  If  the  result 
comes  from  analysis,  it  is  necessary  to  identify  the  thread,  for  which  a  tag  is  needed. 

The  above  analysis  example  started  with  an  analysis  question,  accessing  a  result  that  already 
existed,  then  drilling  down  into  Levels  2  and  1.  Because  of  the  result  accessed,  the  drilling  down 
began  with  looking  for  instances  of  TCT  and  FTP  on  the  same  platform.  The  example  ignored 
the  fact  that  the  result  was  already  present,  and  that  thread  paths  already  existed,  in  order  to 
illustrate  the  analysis  process. 

2.5  Level-2:  System  Information  Tagging 

Analysis  begins  with  a  question,  then  assembling  the  appropriate  tags  to  pull  the  thread.  Assum¬ 
ing  that  one  wishes  to  generate  new  results,  the  pull  will  be  from  Levels-2  and  1.  The  following 
two  sections  describe  the  tagging  schemes  for  these  two  levels. 

Level-2  will  contain  much  context  meta-data.  Examples  are: 

Event  identifier 

Operation  or  MESL  within  the  event 

Type  of  operation  being  examined 

Description  of  the  specific  C3I  structure 

Descriptions  of  the  specific  hardware  and  software  systems 

With  each  datum  in  Level-2  there  will  be  tags  identifying  the  associated  meta-data.  Level-2 
information  deals  with  system  capabilities.  A  example  of  defining  a  system  was  presented  above 
using  generic  sensor  as  the  example.  It  illustrates  the  many  subsystems  that  make  up  the  total 
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system.  Level-2  information  can  be  for  a  system,  subsystem,  or  combination  of  subsystems,  as 
illustrated  in  the  following  diagram.  Also  illustrated  is  that  archived  information  can  be  a  subjec¬ 
tive  “opinion”  or  can  be  a  “result”  pulled  from  Level- 1  data  (Figure  2). 

Recall  that  the  data  tagging  structure  has 
three  categories:  Things,  Attributes,  and 
Actions.  For  Level-2  information, 
actually  for  all  data,  the  Things  category 
tagging  is  natural.  It  consists  of  identify¬ 
ing  the  system  itself  and  those  things  on 
which  it  sits.  Attributes  is  also  natural. 
The  information  can  be  a  status  report 
made  at  a  particular  time,  what  the 
mission  is,  the  workflow,  etc.,  and  it  can 
include  more  than  one  attribute  in  a 
single  data  entry.  Actions  at  this  level  are 
more  subtle  as  a  category.  At  Level-2, 
Actions  refers  to  information  about  system  performance  when  an  Action  is  being  performed. 
Reporting  on  the  status  of  a  communication  system  might  be  that  it  is  down.  Reporting  on  its 
Action  might  be  that  the  data  rate  is  too  slow  for  a  particular  peak  load.  Such  information  can 
be  time  marked,  can  refer  to  a  time  period,  or  may  have  no  time  associated  with  it  being  a 
general  capability  comment. 

2.6  Level-1 :  Data  Tagging 

The  distinctive  characteristic  of  Level- 1  data  is  that  it  contains  events  that  occur  at  a  specific 
time.  Event  data  can  be  subjective  or  objective.  Examples  are: 

Objective:  a  target  folder  being  sent  to  the  fires  cell,  or  a  STOW  simulation  target  inject 
Subjective:  an  observation  or  an  opinion,  such  as  an  assessment  node  becoming  overloaded 

Subjective  opinions  are  needed  in  Level- 1.  An  example  shows  the  importance  of  doing  so.  Take 
the  case  of  an  observation  that  an  assessment  node  is  overloaded.  There  may  be  available  for  that 
time  period  objective  data  that  three  sensor  hits  arrived  at  the  node  within  a  five  minute  period. 
Combining  the  subjective  and  objective  data  allows  one  to  draw  the  conclusion  that  this  node 
becomes  overloaded  if  more  than  two  targets  are  to  be  processed  within  a  10  min  time  period. 
This  conclusion  could  be  a  Level-2  datum  entry  for  the  system. 

There  is  little  difference  in  the  tagging  for  Levels- 1  and  -2.  Event  time  is  unique  to  Level- 1,  and 
Level- 1  will  always  deal  with  an  action,  and  be  so  tagged,  while  most  information  in  Level-2  does 
not. 

A  better  understanding  of  tagging  at  the  two  lower  Levels  of  data/information  can  be  obtained 
through  examples.  The  sensor  system  is  again  the  example.  Thus,  the  “sensor  system”  tag  is 
understood  to  be  attached.  We  only  list  the  specific  tags  for  that  data,  not  all  the  context  tags, 


L-2  Opinion 


Sub-System  A  |  Sub-System  B  |  Sub-System  C  |  Etc. 


I  V3 

I  L-2  Result 

i 

i 

! -  System - 

Criteria  1 :  Correspondence  between  L-2  questions  and  L-2  information 
Criteria  2:  Logical  relationship  between  L-2  information  and  L-1  tags 

Figure  2.  Level  2  opinions  concerning  the  performance  of  a 
particular  system  as  supported  by  L-1  data. 
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such  as  platform  and  sensor  type. 


Level- 1  data: 


Level-2  info: 


Data  or  Information 

Time  to  create  folder 
Time  to  mensurate 

Target  info  transmission 

Time  for  weapon/target  pairing 

“The  fires  cell  configuration 
significantly  reduces  the  TCT 
timeline  when  compared  to  a 
baseline  configuration.” 


Partial  Tags 

Decide,  GISRS-M  Terminal 
Decide,  PTW+  Terminal,  Target 
Type,  Physical  Environment 
Information,  Target-Information, 
Transmit,  E-mail 
Decide,  LAWS  Terminal, 

Fires  Cell  configuration  reference 

TCT,  Latency,  COP(?), 

Fires  Cell  configuration  reference, 
person  entering  opinion  or 
analysis  thread  reference 


This  conclusion  could  have  been  produced  directly  by  an  observer  or  by  accessing  the  noted 
Level- 1  data.  If  it  came  from  the  Level- 1  data,  perhaps  from  that  data  referred  to  in  this  example, 
then  a  better  Level-2  statement  and  tagging  would  be: 

Level-2  info:  “The  use  of  GISRS-M,  PTW+,  TCT,  Latency,  JFE  Cell, 

and  LAWS  in  a  JFE  Cell  LAWS,  GISRS-M,  PTW+ 

configuration  improved  analysis  thread  reference 

the  TCT  timeline.” 


The  following  is  a  constructed  example  of  Level-2  information  at  the  subsystem  level. 

Level-2  info:  “TARPS-CD  imagery  did  not  TARPS-CD,  Detect,  Target  Type, 

have  sufficient  resolution”  TCT,  Environment,  Location 

If  the  observer  logged  a  time  at  which  this  observation  was  recorded,  it  could  be  possible  to 
correlate  it  with  Level-1  data  concerning  an  actual  target,  sensor  status,  etc. 


3.0  EQKMS  Methodology 

A  key  facet  of  the  knowledge  management  process  is  the  organization  of  information  to  enable 
various  levels  of  searching  and  indexing.  This  includes  the  creation  of  metadata  to  summarize 
and  categorize  data.  The  knowledge  repositories  will  contain  output  from  various  processing 
algorithms  including  qualitative,  relational,  neural,  agent,  and  rule-based.  Once  in  a  uniform 
structure,  the  various  search  and  retrieval  methodologies  can  be  applied  against  the  repository  to 
provide  perspective  within  and  across  FBEs.  Categorizations  of  qualitative  resources  specific  to 
the  needs  of  IJWA  analysis,  and  implemented  in  the  Ethnographic  Qualitative  Knowledge 
Management  System  (EQKMS),  address  information  sources  as  variables  in  search  scenarios: 
either  high-level  from  decisions  to  specific  events  or  vice-versa. 
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Present  levels  of  analysis  are  addressing  variables  such  as  decision-making  processes  in  Fires, 
discussions  of  data  flows  in  critical  systems,  observations  of  command  officers  and  pertinent 
post-FBE  analysis,  human-in-the-loop  operations  at  various  nodes,  output  from  electronic 
workstations,  etc.  As  the  knowledge  repository  builds,  with  input  from  various  FBEs,  the  system 
will  support  pattern  recognition  and  matching  across  FBEs. 

An  objective  of  the  IJWA  EQKMS  project  is  to  support  FBE  analysis  using  classification  schemas 
which  enable  high-level  data  extraction  to  support  multi-level  reporting,  including  the  capability 
to  drill-down  from  high-level  decisions  to  supporting  data,  and  to  expand-up  from  event  data  to 
actions  and  results.  Analysis  in  support  of  this  process  will  provide  a  continual  stage  of  synthesis 
into  high  level  results  which  assess  overall  operational  efficiency.  This  analysis  is  derived  from 
data  mining  into  the  knowledge  repository  and  the  application  of  various  knowledge  manage¬ 
ment  and  artificial  intelligence  tools  to  extract  information  from  the  system.  Perhaps  the  stron¬ 
gest  rationale  for  this  approach  is  that  the  results  will  contain  the  thread  that  generated  the  result. 
The  original  data  and  context  will  not  be  lost — which  is  a  problem  common  to  other  method¬ 
ologies.  Thus,  if  needed,  the  thread  can  be  easily  traced  to  source  data  and  the  result,  decision  or 
analysis  evaluated  in  the  original  context  in  which  it  occurred. 

3.1  Data  Capture  Variables 

Many  types  of  data  are  required  to  successfully  analyze  FBEs.  The  major  categorizations  of 
qualitative  and  quantitative  information,  and  the  sources  of  the  information,  are  identified 
within  Figure  3. 

The  first  two  data  types  listed  deal  with  specific  events  that  occur  at  specific  times.  The  next  two 
deal  with  systems  and  processes,  a  step  higher  in  abstraction  and  synthesis.  The  last  two  deal 
with  the  results  of  the  various  operations  within  the  experiment.  In  order  to  advance  analysis  in 
these  areas,  within  the  context  and  with  the  capabilities  as  described  above,  a  methodology  for 
information  organization  was  developed. 

3.2  Systems  Organization 

Required  attributes  for  the  EQKMS  include  speed,  access  to  a  wide  variety  of  qualitative  and 
quantitative  information  sources,  and  extraction  efficiency.  Ideally,  the  methodology  should 


y 

Experiment  Events 

Process  capabilities 

•  Observer  data  logs 

•  Expert  and  operator  opinions 

•  Operator  logs 

•  Post  experiment  interviews 

•  Simulation  injects 

*  Electronic  data 

•  Electronic  data 

Evaluation  of  experiment  results 

/ 

Information  throughput 

•  Operator  opinions 

•  Electronic  data 

Analysis 

/ 

System  and  sub-system  performance 

✓ 

Experiment  results  implementations 

•  Expert  opinions 

•  Post  experiment  interviews 

•  Electronic  data 

•  Interviews  with  the  Fleet 

Figure  3.  Primary  qualitative  and  quantitative  data  categories. 
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support  data  and  information 
extraction  from  source  systems. 
In  other  words,  the  original 
processing,  reporting,  and  other 
output  capabilities  of  the  source 
systems  should  be  supported. 

For  situations  requiring  contex¬ 
tual  searching  or  mapping  across 
loosely  connected  variables,  the 
methodology  should  enable 
human  knowledge  managers  to 
quickly  scan,  associate,  and  link 
pertinent  events.  FEE  analysis 
would  thereby  be  supported 
through: 

(a)  the  archiving  of  information 
from  many  events  so  that  synthe¬ 
sis  can  be  easily  done  across  events; 

(b)  the  efficient  extraction  of  information  through  an  organization  of  information  into 
topical  areas,  thereby  speeding  the  analysis  processes  and  providing  an  avenue  for  search  and 
retrieval  linkages  across  systems  and  resources; 

(c)  a  methodology  for  efficient  correlation  of  data  across  parameters,  thereby  enabling  the 
development  of  knowledge  bases  and  extraction  processes  which  would  be  difficult  for 
analysts  to  develop  on  their  own. 

An  initial  application  of  the  above  involved  the  creation  ofTCT  timelines  using  target  numbers 
to  track  events  and  decisions  specific  to  targeting  (Figures  4  &  5).  IRC  chat  information  was 
synchronized  using  target  numbers  and  these  numbers  were  searched  across  the  chats.  In  the 
near  future  it  will  be  possible  to  pull  information  from  the  observer  data-logging  sheets  in  the 
same  manner.  This  addition  will  provide  context  to  issues  faced  by  operators  in  the  FBEs.  A 
typical  search  of  a  target  number  will  reveal  all  operator  activities  regarding  that  target.  With 
additional  knowledge  processing  the  target 
numbers  can  be  linked  back  to  the  original 
sensor  information.  The  linkage  of  the 
qualitative  with  the  quantitative  will 
produce  specific  events  and  associated 
actions.  The  inclusion  of  the  observer  data 
logs  will  provide  an  objective  assessment  to 
include  events,  operator  actions,  and  any 
additional  variables  which  may  influence 
the  processes. 

Thus,  the  data  structure  organization 
methodology  will  allow  one  to  construct 
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Figure  5.  Example  information  flow  for  qualitative  analysis. 
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*  Data  Archiving  /  Warehousing 

-  Linear  Analysis 

-  Event  and  system  information 

-  Decisions  and  processes 

-  Traceable  threads 

-  Drill  up/down 


►  Multidimensional  analysis 
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-  Human- machine  data 
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LAWS  PTW 
G1SRS  COP 
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STOW 

Command 

Compass 

JFACC/JFMCC 


Engagement 
Actions  References 
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Figure  4.  TCT  qualitative  analysis  process. 
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data  extraction  threads  that  address  specific  operational  and  system  questions.  The  data  extrac¬ 
tion  methods  may  be  specific  to  questions  unique  to  a  particular  FBE,  and  in  the  future,  across 
FBEs.  In  a  projected  application,  given  adequate  data  input  and  processing  resource  organiza¬ 
tion  before  an  experiment,  and  given  that  proper  data  can  be  entered  into  the  system  quickly 
enough,  the  KM  system  could  help  in  assembling  the  Final  Report.  A  long  term  objective  would 
be  to  provide  analysis  and  assessment  with  increasing  speed  and  efficiency.  The  tools  and  ap¬ 
proaches  being  developed  in  the  EQKMS  will  enable  this  evolution.  An  external  variable  not 
within  the  control  of  the  EQKMS  is  the  data  organization  structure  imposed  by  systems 
throughout  the  FBE.  To  the  extent  that  standardized  data  and  information  organization  struc¬ 
tures  can  adopted  by  all  systems  and  users  then  the  information  can  be  ever  more  quickly  synthe¬ 
sized  and  integrated  into  the  analysis  process — and  perhaps  one  day  into  decision  processes.  The 
EQKMS  is  utilizing  a  methodology  to  support  such  data  and  information  organization. 


4.0  Knowledge  Hierarchies 

Qualitative  information  is  being  organized,  tagged  and  labeled  following  methodologies  devel¬ 
oped  on  two  levels.  The  first  level  addresses  the  coding,  tagging,  and  detailed  organization.  The 
second  addresses  the  means  through  which  various  systems  can  organize  their  information  and 
data  to  assist  in  detailed  extraction  across  computer-based  systems. 

4.1  Decision  and  Event  Extraction 

Data  archiving  to  support  decision  and  event  extraction  consists  0/  tagging  and  coding  based  on 
a  methodology  which  can  be  applied  across  information  resources.  This  information  is  catego¬ 
rized  on  the  3  levels  discussed  earlier.  Level  1  organization  considers  objective  and  subjective 
data  that  directly  address  events.  As  discussed  earlier,  Figure  1  illustrates  the  relationship  of 
“things”  to  “attributes”  and  “actions”  in  the  EQKMS  schema.  This  simple  categorization  enables 
the  data  coders  to  seek  existing  relationships  in  the  data  and  uncover  mixtures  of  variables  which 
provide  insight  on  the  issues  being  investigated.  A  ready  example  is  the  categorizations  of  the 
“process”  issues  in  IRC  wherein  one  of  the  data  coders  spotted  problems  that  operators  were 
having  with  communications — both  human-to-human  and  human-to-machine — and  began 
inputting  process  classifications  into  EQKMS. 

A  second  level  of  analysis  follows  the  illustration  in  Figure  2.  Level  2  analysis  concerns  the 
performance  of  a  particular  system  and  interfaces  with  Level  1  analysis  to  discover  relationships 
between  the  application  of  a  particular  technology  and  objective  feedback  on  the  processes 
invoked  during  the  application — whether  human  or  technological.  This  analysis  is  reflected  in 
the  “process”  issues  in  EQKMS.  When  the  live  operator  Level  2  discussions  are  coupled  with  the 
corresponding  assessment  by  the  observers  in  the  data  sheets  there  is  a  strong  basis  for  objective 
assessment  of  a  particular  system  or  operation. 

Level  3  analysis  builds  on  the  Levels  1  and  2  to  present  conclusions  that  address  system  capabili¬ 
ties  at  the  initiative  level.  At  this  level  various  decision  methodologies  can  be  applied  against  the 
Level  1  and  2  data  to  arrive  at  high-level  analysis.  Analysis  and  representation  categorizations 
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include  the  following: 


Robust  decision  analysis  and  support  information  utilizing  structures  supporting 
linear,  multivariate,  multidimensional  analysis 

Multiple  user  relevance,  supporting  non-projected  and  unanticipated  environments 
and  circumstances 

Schemas  which  represent  useful  knowledge  categorization  and  provide  frameworks 
which  stimulate  questions  and  insights 

Knowledge  bases  to  better  understand  systems  and  processes,  and  to  identify  usage 
and  implementation  patterns  and  problems 

Conclusions  addressing  capabilities  in:  Time  Critical  Strikes  and  Fires;  Theatre 
Ballistic  Missile  Defense;  Common  Operating  Picture;  etc. 


Within  EQKMS  the  analysis  above  is  supported  through  data  mining  into  the  knowledge  base  to 
produce  relevant  tags  and  process  information  linked  to  the  source  data.  Plus,  the  search  pro¬ 
duces  a  thread  back  to  the  original  information  and  data  for  in-depth  analysis.  An  added  value  is 
that  the  qualitative  search  technique  produces  the  result  within  the  original  context. 

4.2  Cross-System  Organization 

Linear  and  multidimensional  analysis,  within  the  context  of  the  EQKMS  distributed  qualitative 
knowledge  system,  would  typically  begin  with  a  specific  event  or  decision.  For  example,  the 
identification  of  a  target,  process  or  system.  As  a  first  step  the  variable  would  be  traced  from 
inception  to  conclusion.  Event,  decision  and  time  variables  would  be  tracked  in  a  linear  fashion. 
Qualitative  multidimensional  analysis  extends  the  scope  of  the  inquiry  to  bridge  parallel,  adja¬ 
cent  and  complementary  information  resources.  For  example,  an  event  or  decision  can  be 
tracked  from  inception  to  conclusion  through  linear  analysis,  and  through  multidimensional 
analysis  this  information  can  be  supplemented  with  additional  perspectives  and  supporting 
data — such  as  observer  logs,  independent  assessments,  post-FBE  analysis,  data  from  other  sys¬ 
tems,  etc.  Time,  event  and  decision  data  provide  the  common  link  between  the  various  re¬ 
sources.  The  end  result  supports  high-level  and  low-level  analysis  across  time,  and  linear  and 
multidimensional  assessments  of  event  and  decision  variables. 

In  addition,  the  above  variables  may  be  cross-referenced  to  the  human  analysis  taking  place 
within  IJWA  to  further  assist  in  the  assessment  of  information  flows,  decision  processes,  and 
systems  issues.  For  example,  sensor  data  may  be  linked  to  an  event — and  both  linked  to  human 
actions  at  each  step  in  the  decision  process— with  the  collective  linked  to  various  levels  of  analy¬ 
sis  (both  real-time  and  post-FBE).  In  the  post-FBE  analysis,  the  IJWA  coding  and  tagging 
process  itself  produces  a  level  of  analysis  as  the  trained  knowledge  workers  spot  abnormalities  and 
deviations  in  FBE  systems  and  processes.  Such  high-level  analysis  is  not  available  without 
human  processing  of  the  information.  Unexpected  results  can  be  identified  and  tagged  as  inter¬ 
vening  or  unforeseen  variables.  An  example  would  be  a  systems  process,  a  human  communica¬ 
tion  error,  or  an  assessment  of  delays  or  time  differentials. 

The  key  to  synchronizing  these  various  systems  and  levels  of  analysis  resides  in  the  knowledge 
hierarchies — which  are  common  across  information  sites,  and  as  feasible,  across  data-generating 
units.  A  common  information  organization  structure  (Figure  6)  will  enable  diverse  systems  and 
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Figure  6.  EQKMS  data  and  information  organization  process. 


distributed  operations  to  make 
their  information  available  in  a 
format  that  lends  itself  to 
analysis  across  FBE  operations. 
The  long-range  implications  of  a 
uniform  information  classifica¬ 
tion  hierarchy  are  considerable 
and  at  the  basis  of  the  EQKMS 
methodology.  Specifically,  the 
system  is  being  designed  to 
support  various  levels  of  analysis 
within  an  open,  distributed, 

[and  as  feasible]  object-oriented 
infrastructure. 


This  methodology  can  be  contrasted  with  a  singular  or  proprietary  approach  wherein  all  data 
must  be  processed  within  a  single  methodology  or  architecture — -which  is  generally  costly  and 
time-consuming.  Rather,  the  EQKMS  methodology  accepts  data  from  whatever  source  and 
location  and  organizes  the  information  for  access  and  data  mining 
via  various  techniques  and  approaches.  It  does  not  require  that  all 
information  be  loaded  into  a  central  service  but  instead  utilizes  data 
and  information  in  its  native  format  and,  as  feasible,  applies  various 
search,  retrieval  and  processing  methodologies  to  the  data.  This 
methodology  thereby  utilizes  existing  processes  and  personnel  and 
does  not  require  an  extensive  recoding  or  reformatting  of  data,  only 
the  processing  of  the  source  data.  This  processing  may  include  the 
tagging  of  pertinent  information  (Figure  7),  the  generation  of 
metadata  or  meta-indexes,  full  text  and  object  or  relational 
searches,  and  knowledge  or  database  loading  of  either  the  core 
material  or  the  metadata  derived  from  that  material.  Metadata 
indexes  will  enable  a  fast  and  robust  search  and  retrieval  of  both 
processed  and  unprocessed  information  (via  searches)  for  output  to 
reports,  web-based  display,  or  transfer. 

4.3  Ethnographic  Processes 

The  Ethnographic  Qualitative  Knowledge  Management  System 
(EQKMS)  facilitates  the  analysis  of  qualitative  data  collected  for 
both  qualitative  and  quantitative  research.  The  process  includes 
formatting  the  data,  coding  data  as  a  means  of  noticing  interesting 
information,  tagging  relevant  documents  or  phrases,  and  entering 
code  words  to  assist  in  the  filtering  and  extraction  of  knowledge. 

As  an  example,  in  FBE  Golf,  EQKMS  processing  of  the  IP  Chat 
files  proved  to  be  an  efficient  means  for  contextual  analysis  of  both 
operator-specific  and  environmental  variables.  1  he  qualitative  data  variables  applied  to  files. 
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was  primarily  textual  and  included  materials  such  as  interview  transcripts,  field  notes,  open- 
ended  survey  responses,  chat  files,  and  various  reference  documents.  The  knowledge  processing 
cycle  within  EQKMS  is  itself  a  form  of  high-level  analysis,  with  the  knowledge  workers  noticing 
interesting  relationships  within  the  data,  marking  pertinent  phrases  with  code  words,  and  inte¬ 
grating  variables  across  resources  during  the  search  processes.  Within  EQKMS,  the  IJWA 
analytic  scheme  for  noticing  relationships  in  the  data  can  be  as  simple  or  as  complex  as  is  re¬ 
quired.  IJWA  can  also  revise  and  refine  these  schemes  as  the  work  progresses. 

EQKMS  is  thereby  a  collection  of  procedures  designed  to  enhance  the  process  of  qualitative  data 
analysis.  Although  there  are  varying  perspectives  on  how  researchers  should  conduct  such 
activities,  the  essence  of  qualitative  data  analysis  almost  always  involves  the  process  of  noticing, 
collecting,  and  thinking  about  things  that  are  interesting  within  the  data.  With  the  FBE  Golf 
information  this  process  has  allowed  IJWA  to  put  not  only  a  large  amount  of  data  into  the 
knowledge  base  but  also  information  from  various  perspectives — 1st,  2nd  and  3rd  party.  Once 
imported  into  the  EQKMS,  data  files  are  in  a  40  character  line,  hanging  indent  format.  This 
makes  the  files  easier  to  read,  code  and  organize  compared  to  non-qualitative  systems. 

The  process  of  code  mapping  consists  of  reading,  rereading  and  noticing  interesting  events  in  the 
data  and  then  marking  those  things  for  later  retrieval.  Code  mapping  is  a  means  of  sketching 
out  the  analytical  scheme  of  the  data,  as  well  as  a  step  that  allows  the  searching  and  filtration  of 
information  by  these  code  words.  EQKMS  Phase  1  employs  a  code  book  as  a  type  of  meta¬ 
index.  It  references  the  code  words  attached  to  the  data  file.  As  code  words  are  added  to  the 
files  they  are  automatically  added  to  the  code  book.  The  code  book  contains  information  about 
each  code  word,  and  thereby  about  each  item  in  each  of  the  files — including  the  parent  code 
associated  with  the  code  word  (if  there  is  one);  the  nature  of  the  information  being  coded  (e.g., 
whether  or  not  the  code  word  defines  text);  definitions  associated  with  the  code  words;  the  date 
the  code  word  was  entered  into  the  code  book;  and  the  last  date  the  code  book  was  modified. 
The  code  book  also  allows  for  the  organization  of  the  codes  into  hierarchical  groups  called 
families  (similar  to  trees  in  object-oriented  systems),  and  the  ability  to  view  the  code  book  as  a 
list  or  a  family  tree  depicting  the  relationships  among  parent  and  child  codes. 

While  the  analyst  is  working  with  and  thinking  about  the  data  they  can  use  the  EQKMS  Memo 
function  to  record  ideas,  hunches  and  thoughts  about  the  data  from  almost  anywhere  in  the 
system.  There  are  three  types  of  memos:  project  memos,  file  memos,  and  text  memos.  Each 
memo  can  be  up  to  32  pages  in  length.  Project  memos  are  memos  about  an  entire  project  (e.g., 
IP  Chat  or  FBE  Golf).  A  project  may  have  up  to  1000  memos  attached  to  it.  File  memos  are 
memos  about  an  entire  file.  Each  file  in  a  project  can  have  up  to  1000  memos  attached  to  it. 

Text  memos  refer  to  specific  sets  of  lines  within  a  data  file,  and  each  line  may  have  up  to  26 
memos.  In  addition  to  the  three  standard  memo  types,  EQKMS  Phase  1  allows  the  analyst  to 
create  an  unlimited  number  of  memo  types. 

The  knowledge  workers  coding  the  FBE  GOLF  IRC  files  noted  that  it  was  beneficial  to  use  the 
EQKMS  speaker/identifier  code  features.  This  provides  the  coder  with  the  ability  to  identify  the 
current  speaker/source  within  the  file.  Because  identifiers  can  be  used  to  represent  more  than 
just  speakers,  this  code  is  frequently  referred  to  simply  as  an  “identifier”.  Identifiers  are  defacto 
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code  words  which  can  be  searched  and  filtered.  This  feature  can  be  used  when  querying  or  data 
mining  into  the  EQKMS.  For  example,  if  the  analyst  would  like  to  see  all  data  segments  entered 
by  a  particular  chat  participant  in  relation  to  a  specific  target  this  could  be  filtered  out  of  the 
EQKMS  by  searching  on  the  participants  name  or  call  sign  and  the  target  number.  Within  the 
EQKMS  Face  Sheets  and  Identifier  Sheets  are  lists  of  variables  that  act  as  filters  during  searches. 
Both  Face  Sheets  and  Identifier  Sheets  are  based  on  templates  that  are  created  by  the  EQKMS 
analyst. 

The  template  is  a  list  of  up  to  40  variables  (e.g.,  target_num,  munitions,  sensor)  that  are  the  basis 
of  Face  and  Identifier  sheets.  A  Face  Sheet  is  thereby  a  set  of  information  about  a  whole  data  file. 
It  lets  the  analyst  restrict  a  search  to  data  files  that  meet  conditions  specified  on  the  Face  Sheet. 
An  identifier  sheet  is  a  set  of  information  about  a  speaker  or  a  section  within  a  data  file.  It 
enables  the  analysis  to  limit  a  search  to  segments  containing  identifiers  that  meet  conditions 
specified  on  the  Identifier  Sheet.  For  example,  in  a  chat  file  with  multiple  participants,  Identifier 
Sheets  could  be  used  to  record  the  rank,  name  and  location  of  each  speaker. 

Search  Filters  are  a  means  to  restrict  searches  to  segments  that  meet  specified  conditions.  A  search 
can  be  filtered  using  1)  Face  Sheets,  2)  Identifier  Sheets,  3)  Identifiers,  or  4)  File  Codes.  Combi¬ 
nations  of  filter  types  are  supported.  The  search  procedure  itself  offers  several  options  for  analyz¬ 
ing  data,  including  single  code  searches,  multiple  code  searches  and  identifier  searches.  A  single 
code  search  scans  the  data  for  text  segments  identified  by  a  code  word.  A  multiple  code  search 
locates  text  segments  identified  by  two  or  more  code  words  linked  by  AND  and  NOT.  For 
example,  a  search  can  be  conducted  for  all  segments  identified  by  CODEa  AND  CODEb  but 
NOT  CODEc.  An  Identifier  search  looks  for  text  associated  with  Speaker/Section  Identifiers  in 
the  data  file.  The  text  associated  with  an  identifier  includes  all  the  text  from  the  identifier  down 
to  the  next  identifier  in  the  data  file. 

The  EQKMS  also  has  several  quantitative  outputs  associated  with  a  qualitative  search,  including 
number  of  search  segments  located,  frequency  of  each  occurrence,  and  summary  data  based  on 
the  occurrence  of  associated  items  such  as  memos.  A  segment  output  will  display  the  text  de¬ 
fined  by  a  code  word  and  associated  cross-referenced  information.  Frequency  output  displays  the 
numerical  counts  of  coded  segments  and  calculates  relative  frequency  percentages  for  code 
words — both  within  and  across  data  files.  A  summary  output  lists  the  line  number  coordinates 
of  segments  and  information  such  as  the  size  of  a  segment  (lines  or  pertinent  information). 

When  memo  output  is  selected  the  EQKMS  displays  the  memo  attached  by  the  knowledge 
worker  or  analysis  to  a  particular  file  or  group  of  files. 

4.4  Ethnographic  Application  of  Methodology 

The  IJWA  assessment  of  FBE-Golf  data  included  the  integration  of  IP  Chat  into  EQKMS.  Files 
were  categorized  and  coded  based  on  knowledge  hierarchies  (functional  decompositions)  repre¬ 
senting  various  operational  levels  and  decision-making  processes.  These  ranged  from  command 
to  operational  decisions,  and  from  system-based  actions  to  individual  interpretation.  The  subject 
area  involved  targeting  processes:  from  initial  sensor  readings,  to  target  number  assignment,  to 
the  addressing  and  resolution  of  the  target.  The  EQKMS  process  captured  the  qualitative  data  in 
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the  targeting  process — in  this  instance,  the  discussion,  opinions,  systems  data/references  and 
commands. 

Chat  files  are  the  means  through  which  the  participants  in  the  experiment  coordinate  dynamic 
elements  in  the  FBE.  The  different  chat  files  address  different  aspects  of  the  operations.  The 
focus  of  the  EQKMS  analysis  is  to  structure  the  topics  under  discussion  to  provide  high-level 
analysis  of  the  dynamics  of  the  activities  and  interactions.  For  example,  chat  regarding  the  actual 
engagement  and  resolution  of  targets  is  mostly  found  in  the  ENGAGEMENT  area,  and  sensor 
events  are  more  commonly  discussed  in  the  GISRC,  SENSOR  PLANNING,  and  TARGETS- 
SENSORS  chats.  The  EQKMS  analysis  process  utilizes  the  knowledge  management  hierarchies 
to  categorize,  filter  and  search  this  data.  Organizational  hierarchies  implemented  through  the 
chats  address  platforms,  operators  on  these  platforms,  and  command  level  decisions.  Also  ad¬ 
dressed  are  chat  topics  discussing  the  overall  cohesion  of  the  FBE  and  the  common  operational 
picture. 

Worthy  of  note  is  that  often  data  that  appears  as  though  it  should  be  limited  to  a  specific  chat 
will  be  found  in  files  where  it  is  not  expected.  For  example,  Battle  Damage  Assessment  (BDA) 
would  be  expected  to  be  found  in  the  BDA  chats  but  is  most  commonly  discussed  in  the  EN¬ 
GAGEMENT  chats.  Many  command  issues  that  would  be  expected  to  be  in  the  COMMAND 
chat  are  in  the  ENGAGEMENT  orJECG.CMD  chat  files.  The  EQKMS  allows  not  only  the 
organization  but  the  filtration  of  information  across  chats,  and  thereby  across  participants. 
Assessments,  filtrations,  and  drill-down  capabilities  can  thereby  work  across,  or  integrate,  data 
sources,  participants,  events,  etc.  A  benefit  of  the  EQKMS  methodology  in  this  process  is  the 
ability  to  extract  relevant  data  pertaining  to  command  and  operational  decisions  and  then  "drill 
down"  into  supporting  insights.  Through  the  searches,  and  the  corresponding  assessment  of  the 
grouped  and  categorized  results,  it  is  possible  to  not  only  assess  the  circumstances,  and  interven¬ 
ing  variables,  but  the  personal  insights  and  perspectives  of  the  participants. 


5.0  Knowledge  Organization  Hierarchies 


Data  Type 


Category 


Command 


The  application  of  the  knowledge  processing  hierarchy  outlined  in  Figure  6  is  applied  to  a 
COMMAND  functional  decomposition  sequence  in  Figure  8.  In  this  application  the  initiative 

is  IRC  and  the  specific  chat  area  is 

initiative:  ip  Chat  identified  as  the  data  type.  The  next 

layer  is  the  category,  in  this  instance 
discussion  of  C2  issues.  Focus  and 
subfocus  areas  delineate  means  to 
organize  data  and  information  of 
increasing  specificity.  There  are 
several  rationales  for  this  approach. 
Foremost  is  standardization  across 
diverse  data  and  information 
sources.  Without  a  common  frame 
of  reference  those  developing  infor- 
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Figure  8.  Application  of  the  information  organization  process  schema  to  a 
data  resource. 
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mation  and  knowledge  systems  will  have  a  difficult  time  integrating  information — and  under¬ 
standing  what  information  another  unit  may  be  processing.  Ideally,  the  various  information  and 
knowledge  systems  might  exchange  information  and  one  system  might  utilize  the  output  of 
another.  In  such  an  application  a  common  hierarchy  of  information  or  categorization  would 
allow  an  easy  reference  and  exchange.  As  resources  are  linked  across  the  networks  these  hierar¬ 
chies  may  allow  information  resources  at  diverse  locations  to  be  interconnected  and  operate  as  a 
cohesive  unit. 

For  example,  sensor  data  is  presently  processed  through  a  number  of  operations,  all  using  propri¬ 
etary  information  storage  and  report  generation  systems.  When  one  system  wishes  to  exchange 
information  with  another  the  data  or  information  must  be  processed  into  the  new  system.  This 
is  often  a  “top  down”  process.  Using  Figure  8  as  an  example,  if  all  operations  utilize  “Command/ 
C2/Sensor/ Allocation”  as  an  information  organization  schema  then  the  process  of  linking  various 
resources  becomes  more  probable,  the  overall  cost  of  information  processing  is  reduced,  the 
speed  of  information  transfer  is  increased,  and  the  likelihood  of  knowledge  searches  producing 
comprehensive  analysis  is  increased. 

An  application  within  IJWA  is  a  case  in  point.  EQKMS  is  an  application  of  the  coding  structure 
discussed  earlier  and  illustrated  in  Figure  6  to  categorize  FBE  qualitative  and  quantitative  data. 
Continuing  the  Figure  8  Command  area  as  an  illustration,  the  information  can  be  organized, 
tagged  and  coded  using  the  hierarchy  as  a  reference.  Various  identification  sheets  can  be  coded. 
The  categorizations  enable  an  increasingly  finer  tuning  of  the  searches.  A  fine  grained  search, 
specific  to  the  subfocus  area  of  interest,  would  produce  linear  knowledge  that  is  narrowly  tuned 
to  the  event,  decision  or  process  of  interest.  An 
example  would  be  a  search  on  a  target  number 
or  a  process  (Figure  9).  The  target  is  addressed 
by  different  operators  in  different  chat  areas, 
with  each  operator  addressing  different  aspects 
of  the  target.  In  this  example  the  discourse  is 
charted  by  time  and  operator.  The  figure  shows 
some  of  the  search  tags  entered  into  the  data 
files — which  open  avenues  for  an  examination 
of  broader  issues  affecting  an  operation.  A 
coarse-grained  search  would  reveal  the  item  of 
interest  and  related  information  as  documented 
in  the  sample,  plus,  related  information.  For 
example,  a  coarse  grained  search  may  draw 
sections  of  post-FBE  interviews,  observer 
insights  from  the  data  sheets,  or  reference 
information  used  to  make  a  decision  or  enact  a 
process  during  an  FBE.  All  of  these  may  be  tied 
to  a  specific  target  or  process  issue. 


SEARCH  RESULTS  7/20/2000  8:28:40  PM 

SEARCH  CODE:  T#GA1700 
SEARCH  TAGS: 

#- TARGET  #-TARGET_NUM  # -DTF  #-GISRC 
#-T#GA1700  #-T#GA1710  #-T#GA1720 


FILE:  #TC0405 

[14:23]  <Greg_IKE_GISRC>  Anz  -  confirm 

jtw  menstration  of  those  target, 
what  target  nums? 

[14:23]  <ANZ_GISRC_9>  gal700,  gal710, 
gal720 . . . 


FILE:  #EN0405 

[14:40]  <DTF__IKE>  anz_laws ,  pis  ask 

anz-gisrc9  if  they  have  established 
DTF  for  ga!700 ,  gal710/  ga!720 


FILE:  #TC0405 

[14:41]  <DTF_IKE>  anz-gisrc9,  have  you 

established  DTF  for  gal700,  gal710, 
gal720.  I  prefer  to  chat  in 
target -sensors  room. 


FILE:  #EN0405 

[14:46]  <ANZ_LAWS>  What  tgt  number  ?  we 
fired  on  GA  1700,1710 


The  COMMAND  files  in  IP  Chat  contain 
mostly  Command  participants  and  some 
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Figure  9.  Partial  IRC  discourse  among  TCT  and 
Engagement  operators  addressing  target,  organized  by 
time,  with  thread  to  original  source  file. 


GISRC  operators.  The  focus  of  this  category  is  to  look  specifically  at  battle  command  and  to 
define  the  command  decision-making  process  from  the  commanders  intent,  to  COA  develop¬ 
ment,  to  execution.  This  category  defines  the  data  on  how  the  commander  makes  time-critical 
decisions  based  on  BDA,  realtime  sensor  feeds,  and  change  to  enemy  COA.  These  categories  are 
based  on  the  trends  developed  from  FBE-G.  COMMAND  is  the  primary  place  for  members  of 
the  experiment  to  contact  the  Battle  Watch  Command  (BWC). 


Common  issues  discussed  in  the  COMMAND  data  are:  the  allocation  of  sensors,  specifically 
UAV  sensors;  specific  targets;  digital  target  folders;  and  target  mensuration.  Participants  include 
the  C6  Forward  Battle  Watch  Officer  (C6FBWO),  the  joint  Force  Maritime  Commander 
(JFMCC),  and  the  COPCI.  The  COMMAND  files  are  short  in  comparison  to  the  ENGAGE¬ 
MENT  chat  files  which  also  hold  much  of  the  same  type  of  data.  The  ENGAGEMENT  chats 
have  many  more  participants  involved  than  the  COMMAND  chats  because  of  the  number  of 
participating  operators.  In  sum,  from  the  COMMAND  data  it  is  possible  to  extract  command 
level  decisions  regarding  specific 
processes  from  sensor  events  and 
targets.  The  ability  to  filter  this 
kind  of  information  within  the 
EQKMS  hierarchy  will  help  to 
show  the  effect  on  the  experiment 
that  the  command  level  decisions 
have  on  the  outcome  (target  resolu¬ 
tion). 
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Figure  10.  Common  Operational  Modeling.  Planning 
and  Simulation  Strategy. 


COMPASS  looks  at  data  from  a  single  system  to  see  how  it  relates  to  the  decision-making 
process.  The  relevant  data  in  COMPASS  (Common  Operational  Modeling,  Planning  and 
Simulation  Strategy)  is  in  relation  to:  Rules  Of  Engagement  (ROE);  Platform  location,  coverage 
and  type  (usually  ABL);  operating  space;  and  the  troubleshooting  of  the  COMPASS  server 
(Figure  10).  Future  EQKMS  activities  in  this  area  will  link  COMPASS  platform  and  target 
information  to  operational  data. 


COPCI 


The  COPCI  hierarchy  decomposes  data  from  the  common  operating  picture  system  (Figure  11). 
This  data  can  be  used  to  look  at  situational  awareness  and  how  it  affects  decision-making.  This 
category  is  based  on  data  from  FBE-G.  COPCI  information  categorizations  address  the  FBE 
situation;  threat/ 
target  activity, 
location,  type  and 
resolution;  target 
verification;  system 
status  and  proce¬ 
dures  for  system 
information  entry; 
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Figure  11.  Common  Operational  Picture  Command 
Intelligence. 
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Folder  (DTF)  establishment  and  maintenance. 

These  topics  are  accessible  through  filtration  and 
search  procedures  within  EQKMS — which 
enables  the  desired  "drill  down”  capability  that  is 
needed  to  access  information  from  various  data 
sources  within  the  IP  Chat  files. 

The  ELINT  category  (Figure  12)  is  structured  to  organize  data  specifically  related  to  ELINT 
signatures.  This  category  will  be  used  to  look  at  data  during  deliberate  planning  and  time  critical 
targeting.  This  data  can  be  related  to  the  target-sensors  category. 

5.1  Engagement 

This  category  looks  at  how  we  can  define  data  during  the  engagement  phase  of 'sensor-weapon 
pairing."  This  category'  is  closely  related  to  sensor  management  or  the  target-sensor  category. 

This  data  becomes  important  when  comparing  engagement  qualitative  data  with  quantitative 
data,  such  as  from  LAWS.  ENGAGEMENT  also  addresses  the  integration  of  joint  fires  with  the 
time  critical  targeting  cvcle.  These  categories  are  derived  from  trends  seen  in  FBE-G  and  FBE- 
H. 

The  IRC  ENGAGEMENT  files  (Figure  13)  serve  as  a  meeting  and  discussion  place  for  operators 
at  the  workstations  aboard  various  platforms  in  the  fleet  battle  experiment.  The  battle  environ¬ 
ment  that  the  operators  deal  with  is  very'  dynamic  and  requires  that  the  members  of  the  engage¬ 
ment  chat  be  able  to  converse  in  realtime  about  the  information  they  are  receiving  from  various 
sources.  These  sources  include  UAVs,  sensors,  other  platforms  and  other  systems.  It  is  a  unique 
environment  in  that  the  operators  have  to  be  able  to  think  and  respond  quickly  to  an  ever 
changing  operational  picture  of  the  battle  field. 

ENGAGEMENT  is  somewhat  different  from  the  other  organizational  hierarchies  because  there 
is  an  element  of  command  structure.  There  were  active  command  participants  in  the  ENGAGE¬ 
MENT  IP  chat— usually  a  LCDR  and/or  JFMCC,  JECG  DIR  or  JFACC.  The  prevalence  of  this 
command  structure  in  ENGAGEMENT  impacts  the  knowledge  management  structure  of  both 
the  IP  Chat  and  the  overall  knowledge  management  system.  Specifically,  in  the  design  of  the 
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Figure  12.  Electronic  Intelligence. 
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EQKMS,  references  to  senior  levei  decision-making  can  be  organized  as  distinct  entities  in  the 
overall  hierarchy  of  the  knowledge  management  processes  and  actions  can  be  filtered  or  searched 
based  on  specific  command  guidance  or  oversight  information.  This  can  provide  a  ’’drill  down" 
from  oversight  or  command  perspectives  into  the  actual  target  decision  and  resolution. 

Other  chat  members  active  in  ENGAGEMENT  include  LAWS  operators,  GISRC  operators, 
and  other  members  of  the  experiment  who  are  involved  in  the  actual  engagement  of  targets.  The 
discussion  is  based  on  the  process  of  eliminating  a  target  and  includes  steps  from  the  injection  of 
a  target  into  LAWS  (the  most  prevalent  system  discussed  in  the  ENGAGEMENT  files)  from 
GISR  or  GCCS  to  the  actual  launch  of  munitions  against  the  target.  Other  topics  include: 
target  information:  mensuration  of  the  targets;  creation/management  of  digital  target  folders: 
management  of  the  information  between  the  various  systems;  engagement  of  the  targets; 
prioritization  of  the  targets;  problems  with  target  information;  different  types  of  munitions  sent 
to  targets;  information  about  munitions  in  flight;  new  Time  Critical  Targets  that  need  to  be 
mensurated  and  launched  on;  and  battle  damage  assessments. 

In  sum,  in  the  knowledge  management  organizational  structure,  the  Ethnographic  Qualitative 
Knowledge  Management  System  provides  detailed  references  to  each  of  the  active  elements  in  the 
ENGAGEMENT  chat.  In  the  knowledge  design,  the  engagement  focus  is  thereby  on  specific 
targets,  and  events  addressing  those  targets,  and  pertinent  command  or  specific  decision  pro¬ 
cesses  of  operators  or  participants.  In  the  knowledge  hierarchies  (functional  decompositions), 
the  ENGAGEMENT  data  follows  the  sensor  and  identification  processes  and  precedes  the 
damage  assessment.  ENGAGEMENT  is  thereby  addressed  within  the  EQKMS  as  the  compo¬ 
nent  addressing  target  resolution  and  the  command  and  decision  structures  forming  the  founda¬ 
tion  for  action. 

5.2  FBECOP 

FBECOP  is  for  the  categorization  of  data  from  the  common  operating  picture  system  (Figure 
14).  This  data  can  be  used  to  look  at  situational  awareness  and  how  it  affects  decision-making. 

In  application,  the  FBE  Common  Operational  Picture  chat  area  hosts  GCCS-M  operators  and 
the  technicians  used  to  coordinate  their  efforts.  This  chat  is  crucial  for  keeping  the  COP  up  to 
date  for  GCCS  operators  and  users  throughout  the  experiment.  The  resources  contain  discus¬ 
sions  of  systems  status  and  troubleshooting  of  these  systems.  For  example,  in  FBE-Golf  there 
were  TASID  and  SIPRNET  problems  noted  in  the  FBECOP  chat  areas.  ADVISRT  problems 
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Figure  14.  FBE  Common  Operational  Picture. 
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were  a  topic  of  discussion,  as  was  the  process  of  downloading  and  installing  software  updates. 

Within  the  FBECOP  knowledge  hierarchy  there  is  significant  structure  for  the  recording  of  track 
status  within  the  systems.  The  resolution  of  differences  in  system  status  in  order  to  achieve  a 
synchronized  operational  picture  between  various  platforms  in  an  experiment  is  a  prominent 
theme  throughout  the  FBECOP  hierarchy.  This  would  include  discussion  regarding  specific 
targets,  with  a  recording  of  type,  activity,  location,  resolution,  and  munitions.  The  No  Strike 
Zone  and  Restricted  Fire  Area  overlays  are  discussed  in  the  FBE  data  as  well.  Electronic  Intelli¬ 
gence  (ELINT),  target  information  and  UAV  sensor  information  are  thereby  prominent  infor¬ 
mation  sources  throughout  the  FBECOP  knowledge  hierarchy.  In  IRC,  the  FBECOP  files 
contain  information  concerning  targets  in  early  stages  of  target  nomination. 

5.3  GISRC 


This  category  is  an  attempt  to  look  at  a  specific  system  (GISR)  and  to  derive  some  common 
systems  analysis  from  the  qualitative  data.  The  initial  categorizations  were  derived  from  FBE-G 
IRC  Chat.  Systems  status  is  the  main  focus  of  the  GISRC  (Global  Information  Surveillance 
Reconnaissance  Capability)  data,  -with  sensor  locations  and  missions  as  subfocus  areas.  The  files 
contain  some  tactical  data,  but  consist  primarily  of  GISRC  technical  issues  (Figure  15).  In  an 
FBE,  the  main  purpose  of  the  GISRC  chat  is  for  the  coordination  and  resolution  of  technical 
issues  confronted  during  GISRC  utilization  and  GISRC  data  implementation.  LAWS  and 
GCCS  systems  also  surface  as  a  topic  and  are  thereby  referenced  through  the  EQKMS  knowl¬ 
edge  hierarchy.  Specific  problems  that  are  prevalent  in  the  data  address  systems  software  upgrad¬ 
ing  and  troubleshooting,  TASID  and  ADVSRT  problems,  and  communication  difficulties. 
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Figure  15.  Global  Information  Surveillance  Reconnaisance  Capability 
knowledge  organizational  hierarchy. 


LAWS  1  COP  TASID 


The  coding  scheme 
used  in  the  GISRC 
chat  data  allows 
IJWA  to  do  searches 
char  will  output  the 
segments  of  data 
that  show  the 
processes  of 
troubleshooting  and 

the  areas  that  were  focused  on  during  FBE  operations.  For  example,  when  target  specific  prob¬ 
lems  are  discussed  within  the  GISRC  chat,  this  information  can  be  referenced  through  cross-file 
searches  on  target  numbers,  and  this  information  will  be  filtered  and  made  available  along  with 
the  rest  of  the  data  concerning  that  target.  This  allows  IJWA  to  follow  all  information  concern¬ 
ing  specific  targets  and  the  technologies  addressing  the  targets,  and  in  GISRC,  to  focus  on 
systems  problems  that  may  delay  the  total  resolution  time  of  the  target. 


Also  worthy  of  note  is  that  there  are  times  when  the  GISRC  chat  is  being  used  by  UAVSIM 
operators  for  sensor  allocation  purposes.  The  UAVSIM  participants,  in  some  cases,  were  asked  to 
disconnect  and  move  their  conversations  to  theTARGETS-SENSORS  chat.  This  type  of  cross¬ 
system  analysis  is  a  strength  of  the  IJWA  EQKMS  methodology. 
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Figure  16.  Sensor  data  hierarchy. 
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5.4  Sensors 

The  focus  of  this  category  is  data  related  to  sensor  management  (Figure  16).  Looking  at  the 
FBE-G  and  FBE-H  initial  results,  categories  were  developed  that  would  define  the  relationship 
between  sensors,  commanders  guidance/intent,  and  weapon  systems.  Eventually  this  data  will  be 
used  to  look  at  the  relationships  between  deliberate  ISR  planning,  and  use  of  sensors  during  time 
critical  targeting.  Data  concerning  decisions  on  battlespace  coverage  and  COA  branches  and 
sequels  wiil  be  covered. 

In  application,  the  TARGET-SENSOR  resource  contains  information  utilized  by  operators  in 
charge  of  sensor  allocation  and  target  sensor  data.  Target  information  is  available  as  it  pertains  to 
sensor  readings  and  the  linkage  between  sensor  reports,  systems  reports,  and  target  nomination. 
Supporting  information  is  also  available,  such  as  DTF  data  processing  and  target  type  references, 
location  and  activity. 

Sensor  status,  tasking  and  sensor  BDA  are  documented.  There  is  a  command  element  available 
in  this  resource  and  commander's  guidance  and  command  level  course  of  action  decisions  are 
searchable  in  the  EQKMS  TARGETS-SENSORS  data. 

The  most  common  sensor  events  logged  in  TARGETS-SENSORS  is  UAV  data.  There  is  a 
significant  number  of  data  segments  that  involve  discussions  about  targets  as  registered  by  UAV 
sensors.  The  resultant  mensuration  of  these  targets,  via  the  sensors,  is  available  in  the  knowledge 
base.  Much  of  the  TARGETS-SENSORS  data  is  "pre-target  number"  information,  meaning 
that  the  data  specifies  the  type  of  target  and  the  location  rather  than  the  target  number. 

EQKMS  operators,  by  searching  and  linking  the  target  numbers  to  the  LAWS  data  and  referenc¬ 
ing  TARGETS-SENSORS  data,  can  link  the  target  from  initial  reading  to  conclusion.  The 
coding  scheme  for  these  files  facilitates  the  searching  of  sensor  events  in  relation  to  specific  types 
of  targets,  target  locations,  enemy  missile  launch  events,  and  specific  target  numbers. 

The  J£CG_CMD  knowledge  resources  also  hold  sensor  data,  especially  data  referencing  P-3 
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Figure  17.  Joint  Experiment  Control  Group  Command  data  hierarchy. 


sensors  (Figure  17).  The  location,  allocation,  and  events  surrounding  utilization  of  the  P-3 
sensors  makes  up  the  bulk  of  the  data  in  the  J£CG_CMD  IRC  files.  There  are  also  some  target 
specific  discussions  which  concern  target  type,  location  and  activity,  but  they  do  not  generally 
reference  the  actual  target  numbers.  JECG_CMD  resources  will  be  critical  in  correlating  sensor 
events  (sans  target  numbers)  to  the  actual  target  resolution  process.  The  IJWA  analysts  will  be 
able  to  synchronize  discussions  in  JECG_CMD  andTARGETS-SENSORS  with  LAWS  data 
and  ENGAGEMENT  files  to  recreate  target  events — from  initial  reading  to  identification  and 
resolution. 

5.5  Time  Critical  Strikes  and  Fires 

This  category  focuses  solely  on  the  process  of  time  critical  fires  (Figure  18).  While  related  to  the 
Engagement  and  Target  Sensor  categories;  this  is  an  attempt  to  even  further  isolate  the  fires 
process  with  a  complete  focus  on  time  critical  fires  from  deliberate  planned  fires. 

The  majority  of  the  data  held  in  the  TCSF  files  concerns  systems  status  in  relation  to  the  target¬ 
ing  stations.  Some  of  the  data  is  target-specific.  The  purpose  of  TCSF  IRC  is  to  provide  a  place 
for  guidance  requests  and  CONOPS  (concept  of  operations)  questions.  The  TCSF  chat  area  also 
provides  a  place  and  means  to  facilitate  the  operational  cohesion  of  the  FBE. 
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The  coding  scheme  used  for  the  TCSF  files  enables  the  analysts  to  perform 
searches  that  will  produce:  1)  target  specific  issues,  2)  command  level  decisions 
involving  the  operations,  and  3)  systems  status  and  coordination  data. 

5.6  Effects-Based  Operations 
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EBO  is  an  important  process  for  deliberate  planning.  This  category  is  based  on 
EBO  play  during  the  FBE.  Additionally,  the  subcategories  come  from  EBO  conferences  prepar¬ 
ing  for  Global  2000. 


The  EBO  hierarchy  is  thereby  a  higher-order  classification  schema  which  addresses  overall 
information  and  intelligence  processes  (Figure  19).  This  categorization  contains  macro-level* 
analysis  examining  specific  and  relational  aspects  of  the  FBE.  Foci  such  as  blue  and  red  cell 
activity  can  be  charted  against  strategy  and  implementation  to  provide  a  perspective  on  a  certain 
course  of  action  and  the  results  of  that  action  given  specific  environments  or  variables. 

5.7  Analysis  Processes 

Analysis  is  conducted  during  an  FBE,  immediately  after  an  FBE,  and  during  the  traditional  post- 
FBE  assessment  phases.  Given  the  variety  of  systems  and  processes  in  an  FBE  the  analysis  will 
advance  various  perspectives,  depending  on  the  need,  system  or  sponsor.  The  knowledge  re¬ 
sources  supporting  this  analysis  must  therefore  be  flexible  and  customizable  if  they  are  to  meet 
the  specific  needs  of  the  different  analysts.  In  addition,  the  process  of  sorting,  categorizing,  and 
coding  or  tagging  information  also  provides  a  level  of  analysis  as  the  knowledge  workers  identify 
trends,  deviations,  or  areas  of  interest  in  the  data.  Discussions  about  targeting  and  communica¬ 
tion  difficulties,  mensuration  processes  and  problems,  missile  guidance  issues,  and  IVOX  and 
VAT  communications  issues  would  be  an  example. 
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There  are  also  resources  unique  to  analysis — generally  labeled  as  containing  analysis  information. 
In  some  instances,  such  as  in  the  FBE-Golf  chat  files,  this  data  is  redundant  with  information 
found  elsewhere  but  is  synthesized  into  analysis  files  as  a  means  to  record  significant  events.  This 
channel  may  be  optimally  used  as  a  forum  to  assess  current  situations  within  the  experiment  and 
thereby  provide  an  advanced  level  of  analysis.  This  would  be  useful  for  data  organization, 
filtering  and  searching.  A  benefit  of  EQKMS  in  this  process  is  the  ability  to  extract  relevant  data 
pertaining  to  specific  areas  of  analysis  within  an  experiment,  including  ANALYSIS  chats  as  well 
as  all  other  files  contained  in  IP  Chat  and  developed  throughout  the  experiments.  This  allows  the 
organization  and  filtration  of  information  even  when  the  desired  data  does  not  reside  in  the  files 
in  which  it  is  expected  to  be  logically  associated. 

For  example,  after  coding  and  analysis  of  the  ENGAGEMENT  files  and  the  FBECOP  files  it 
was  apparent  that  there  are  difficulties  in  target  mensuration  for  LAWS  operators.  Once  a  sensor 
detects  a  target,  and  it  is  nominated,  the  target  is  given  a  number  and  this  information  is  added 
to  the  DTF.  In  order  for  a  platform  to  launch  an  attack  on  this  target,  mensuration  must  have 
occurred  in  parallel  with  the  above,  or  nearly  so.  Often  a  platform/workstation  other  than  the 
one  launching  the  missile  will  be  most  appropriate  for  the  mensuration  of  the  target,  and  mensu¬ 
ration  can  be  done  by  several  PTW  workstations  in  an  FBE  so  one  or  more  operators  may  be 
adding  mensuration  to  the  DTF.  The  ANALYSIS  chat  area  would  be  a  viable  resource  for 
discussions  of  this  process  and  an  observer  trained  and  dedicated  to  assessment  of  the  handoff  of 
data  from  sensor  to  mensuration  to  LAWS  may  find  the  ANALYSIS  chat  an  area  appropriate  for 
feedback. 

For  example  (hypothetically),  Anzio,  CSG  and  XVSSN  all  show  target  "GJ3092"  in  their  system. 
XVSSN  has  the  clearest  shot  at  the  target,  but  has  no  ability  to  mensurate  the  target  due  to  their 
position  or  lack  of  direct  access  to  a  sensor.  CSG  has  a  UAV IVO  the  target  in  question  and  can 
mensurate  for  XVSSN.  Often  times  a  process  like  this  occurs  without  a  problem.  However,  it 
seems  very  common  to  have  a  situation  where  mensuration  takes  a  long  time  or  has  problems 
due  to  various  conditions.  Examples  of  these  sorts  of  conditions  would  seem  to  occur  (hypo¬ 
thetically)  when: 

(a)  CSG  is  having  communication  difficulties 

(b)  the  UAV  is  down 

(c)  a  workstation  is  busy  or  the  operator  is  away  from  his/her  station 

(d)  there  is  confusion  about  LAWS  information 

(e)  there  is  a  specific  problem  with  LAWS 

(f)  a  sensor  isn't  available 

(g)  there  are  discrepancies  in  target  numbers 

In  situations  such  as  the  above  the  EQKMS  knowledge  workers  are  able  to  spot  these  problems 
during  the  coding  process  and  mark  them  accordingly.  As  the  EQKMS  project  has  progressed, 
and  the  knowledge  workers  have  become  more  fluent  in  the  processes  and  terminology,  an  added 
benefit  of  this  human  intervention  in  the  processing  of  the  resources  is  the  identification  of 
missing  elements  or  inconsistencies — such  as  the  need  for  a  more  robust  use  of  the  ANALYSIS 
chat  areas,  suggestions  for  usage  of  this  chat,  and  ideas  for  topics  given  missing  elements  in  the 
available  data.  This  form  of  high-level  analysis  is  unique  to  the  qualitative  analysis  process  and 
would  only  be  available  with  the  use  of  knowledge  workers  trained  in  knowledge  extraction, 
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situation  analysis,  and  the  environmental  and 
system  variables  common  to  an  FBE. 

Figure  20  contains  a  segment  from  FBE-G 
illustrating  a  series  of  questions.  The  process 
for  posing  such  questions  itself  provides  an 
interesting  avenue  for  analysis  and  would  be 
appropriate  for  assessment  in  the  ANALYSIS 
chat.  The  recording  of  this  information  in  the 
knowledge  repository,  under  a  number  of 
search  criteria,  would  be  important.  The 
mechanisms  through  which  the  answers  were 
received,  or  lack  thereof,  would  be  an  area  of 
importance  and  unique  to  the  types  of  analysis 
being  conducted  through  EQKMS.  This 
segment  was  tagged  by  an  IJWA  analyst  as 
containing  an  interesting  discourse  appropriate  for  further  assessment. 


FILE:  #TCT0404 

[12:45]  <LCDR_Burian> 

CSG/ANZ/LAS/JFACC/IKE. .NEED  ANSWER  TO 
THE  FOLLOWING  QUESTIONS* - -FROM  THE 
NWDC  LEAD'S  PRESPECTIVE . . . 1 .  DO  YOU 
RCV  UAV  VIDEO/TELEMETRY.  IF  SO, 
NUMBER  OF  CHANNELS  2 .  DOES  YOUR 
GISR  PASS  TGT  NOM  TO  JTW?  3.  CAN 
YOU  DO  A  MANUAL  TGT  NOM  FROM  GISR. 

4.  DOES  GISR  PROPAGATE  TGTS  TO  EACH 
NODE  FOR  BIDDING?  5.  DOES  A  GISR 
TGT  NOM  INJECT  A  TRACK  INTO  THE  COP? 
6.  DOES  A  TGT  NOM  FROM  GISR  POST  AS 
A  DIGITAL  TGT  FOLDER?  7.  DOES  GISR 
PROVIDE  A  UNIT  ID  TO  TGT  NOMS? 

[12:46]  <LCDR_J3urian>  8.  CAN  YOUR  GISR 
NOM  A  TGT  FROM  A  UAV  VIDEO  FEED? 


Figure  20.  Partial  IRC  discourse  through  TCT  with 
implications  for  ANALYSIS  chat  and  post-FBE  knowledge 
organization 


There  are  also  documents  geared  to  the  instruction  and  guidance  of  data  collectors  and  analysts 
in  relation  to  the  collection  of  data  in  the  FBE.  For  example,  the  file  "analytic  framework-FBE 
G  -  Rev  II.doc"  summarizes  initiatives  to  be  examined  during  FBE-G.  It  provides  both  top  level 
and  detailed  questions  of  interest  to  support  operational  analysis  of  the  results  of  the  initiative. 
There  are  questions  for  analysts  to  utilize  concerning  the  COP,  C2,architecture,  Methodology  for 
Sensor- Weapon-Target  Matching,  TCT  Sensor  planning,  loitering  weapons,  Submarine/Special 
Operations  Forces  and  TCT  target  folders. 

Another  file  in  the  Analysis  Planning  folder  is  "Electronic  Data  Collection  in  FBE  G.doc".  This 
file  is  directed  at  data  collection  and  observation  personnel  onboard  the  platforms  in  the  FBE. 
Specifically,  what  electronic  information  needed  to  be  collected  and  how  it  was  to  be  warehoused 
and  transferred.  "FBE-G  Appendix  D  (draft  rev  l).doc"  contains  useful  information  regarding 
the  FBE,  including:  an  overview  of  FBE-G,  Experiment  process,  system  methodologies,  experi¬ 
ment  coordination,  concepts  of  interest,  TCT  analytic  questions/data  collection  instruments, 
LAWS,  GISRC-M,  and  PTW. 


For  post-experiment  information  searching  and  filtration,  the  data  in  these  files  does  not  serve  as 
a  key  resource  for  target  resolution  and  process  issues.  Although  these  files  do  point  to  some 
interesting  questions  regarding  processes  that  have  already  been  defined  through  rigid  coding  and 
search  techniques  within  EQKMS,  the  actual  data  is  not  of  benefit  for  information  extraction. 
Rather,  these  files  may  prove  useful  in  cross-referencing  FBE  search  criteria  once  multiple  FBEs 
are  integrated  into  EQKMS.  The  collective  of  the  FBE  resources  in  the  knowledge  repository 
will  bring  a  greater  understanding  of  the  overall  picture  of  the  experiment  and  the  operational 
efficiencies  of  the  IJWA  analysis. 
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5.8  Data  Sheets 


The  observer  who  authors  the  analysis  in  the  data  sheets  is  an  experienced  participant  in  FBE 
operations  and  is  monitoring  multiple  facets  of  the  experiment  from  a  given  platform.  This  is  an 
indispensable  data  source  for  IJWA's  in-depth  qualitative  analysis  of  the  FBE.  The  observer  is 
able  to  understand  problems  and  issues  that  arise  in  various  areas  as  opposed  to  the  single  opera¬ 
tors  that  are  monitored  and  coded  within  the  IP  Chat  data — and  who  have  a  far  less  broad  view 
of  processes  and  problems.  The  addition  of  the  observer  data  to  IJWA's  knowledge  management 
system  not  only  enhances  search  capabilities  but  fills  in  holes  in  the  data  that  would  be  otherwise 
left  unresolved. 

For  example,  in  the  IP  Chat  engagement  file  for  a  particular  date  there  is  discussion  of  a  civilian 
convoy  being  damaged  in  an  attack.  This  was  originally  coded  as  a  non-military  target  process 
issue  since  civilian  damage  is  obviously  not  a  desired  outcome.  But,  with  the  addition  of  the  data 
sheets  and  observation  files  a  search  reveals  that  an  observer  had  noted  the  civilian  damage  as  a 
misinterpretation  of  data.  This  finding  led  the  IJWA  analysts  to  go  back  to  the  process  labeled  as 
civilian  damage  in  IP  Chat  and  change  the  code  to  a  misinterpretation  of  data  rather  than 
civilian  damage.  This  process  of  misinterpreting  data  will  also  be  reflected  in  the  search  output 
for  the  target  in  question.  Thus,  misunderstandings  can  be  documented  through  the  EQKMS 
search  techniques  and  potentially  embarrassing  environmental  variables  corrected.  Thus,  the 
integration  of  qualitative  and  quantitative  data  sources  across  subject  areas  enables  a  greater 
understanding  of  critical  areas. 

Team  analytic  questions  and  answers  are  also  an  important  part  of  the  data  sheets  resource.  With 
the  integration  of  this  data  into  EQKMS  the  IJWA  analysts  are  empowered  to  extract  some  very 
crucial  information  and  conceptual  relationships.  For  example,  in  a  typical  IJWA  analysis  of  a 
specific  platform,  the  goal  may  be  to  extrapolate  all  targets  that  were  engaged  by  that  platform  on 
a  particular  day  in  the  FBE.  If  searching  only  the  IP  Chat  data  the  search  output  may  be  nebu¬ 
lous  due  to  the  very  large  number  of  participants — all  discussing  many  targets,  some  of  which  are 
not  high  priority,  or  on  duplicate  tracks,  or  are  old  or  from  test  missions. 

Issues  such  as  the  above  will  make  for  a  large  search  output  that  will  require  further  analysis  to 
determine  which  targets,  for  example,  were  engaged  by  Anzio  (given  this  was  the  original  intent 
of  the  search).  The  team  analytic  questions/answers  as  an  independent  information  resource  in 
the  knowledge  base  can  be  searched  and  checked  against  other  information  resources  to  resolve 
unanswered  questions.  A  traditional  search  system  may  inadvertently  filter  key  relationships. 

The  EQKMS  approach  addresses  data  both  independent  of  context  and  within  context  (fully 
integrated  with  all  pertinent  resources). 

Another  example  would  be  search  output  that  correlated  20  or  more  targets  with  a  particular 
platform.  After  the  integration  of  the  data  sheets,  and  analytic  questions/answers  resources,  a 
parallel  search  was  performed  and  the  results  crossed  against  the  IP  Chat  data.  A  discovery  was 
identified  which  might  have  passed  through  a  non-comprehensive  system.  Specifically,  the 
following  passage  appeared  in  the  search  output  of  a  data  sheet: 
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Todays  engagements  totals  32  nominations  with  3  ANZIO  shots  (all  ERGM).  Of 
32  nominations,  only  6  had  imagery  attached  to  the  email.  12  resulted  in  dwell 
expiration,  1  expired  due  to  friendly  SOF  in  the  area,  1  not  high  value,  1 5  lack  of 
mensuration,  and  3  shot. 

A  traditional  search  in  chat  would  not  output  such  specific  summative  information  about  the 
engagements  undertaken  by  Anzio.  The  integration  of  this  kind  of  data  into  the  knowledge 
management  system  also  makes  identification  processes  issues  easier  for  the  analysts  via  parallel 
query  techniques.  Search  procedures  done  only  on  chat  data  do  not  reveal  the  true  complexity  of 
an  engagement.  A  more  accurate  picture  emerges  with  parallel  search  procedures  with  multiple 
data  sources,  and  the  integration  of  quantitative  and  qualitative  data. 

Thus,  the  observers  provide  a  perspective  which  helps  correlate  search  techniques.  Filtering 
processes  derived  from  the  combined  data  makes  it  much  easier  for  IJWA  analysts  to  understand 
where,  when  and  how  relevant  engagements  occurred  and  to  document  pertinent  information 
flows  and  processes.  This  kind  of  understanding  is  unique  to  an  integrated  qualitative/quantita- 
tive  analysis  and  to  the  EQKMS  approach  of  searching  against  data  and  information  resources — 
and  then  comparing  and  contrasting  those  searches  and  findings.  Thus,  EQKMS  brings  a  new, 
higher  level  of  analysis  to  FBEs. 

The  integration  of  this  data  is  also  very  pertinent  to  the  process  issues  that  are  being  explored  by 
IJWA.  For  example,  if  an  IJWA  analyst  thinks,  from  their  interpretation  of  the  data,  that  there 
are  likely  too  many  people  in  the  engagement  chat  and  codes  the  file  as  such,  then  this  assump¬ 
tion  can  be  proven  or  disproven  by  the  additional  coding  of  the  analytic  questions/answers. 
Another  example  of  the  importance  of  cross-information  searches  can  be  shown  by  the  IJWA 
coding  of  each  mention  of  a  communication  link  problem  as  a  specific  process.  In  so  doing, 
IJWA  has  assumed  that  these  communications  difficulties  are  a  problem  to  be  noted  and  perhaps 
addressed  as  reflected  in  the  target  resolution  process. 

Once  the  data  sheet  information  is  coded  and  integrated,  IJWA  can  ascertain  whether  the  team 
analytic  Q&A.  is  viable.  For  example,  if  the  observer  states  that  the  comms  were  down  34%  of 
the  time  on  a  particular  day,  then  this  information  will  allow  IJWA  analysts  to  not  only  verify  the 
problem  with  communications  through  normal  analysis,  but  also  through  correlation  to  the  team 
analytic  report  to  determine  exactly  how  long  the  comms  were  inoperable  in  a  given  FBE.  Thus, 
when  IJWA  searches  processes  associated  with  communications  difficulties  the  output  will 
produce  each  segment  of  data  where  comms  issues  were  occurring  in  realtime  via  the  chat  files 
and  also  the  team  analytic  report  of  exactly  what  percent  of  the  time  the  comms  were  down. 

The  above  process  of  cross-information  clarification/verification  is  also  useful  in  the  defining  of 
different  processes  identified  as  important.  For  example,  the  following  data  segment  from  the 
team  analytic  Q&A  verifies  a  process  that  was  common  throughout  FBE-G  chat. 

A  major  shortcoming  from  the  experiment  is  the  lack  of  COP  input  into  the  GISR 
picture.  It  is  very  difficult  (not  practical)  for  GISR  operator  and  GCCS  operators 
to  look  at  both  systems  and  attempt  to  correlate  tracks  and  sensor  data  via  the  two 
screens. 
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The  above  is  a  reoccurring  issue  within  FBE-G  that  was  identified  by  IJWA  analysts  when  coding 
the  chat  files,  and  further  supported  when  integrating  the  data  sheets  and  observations  informa¬ 
tion  into  the  knowledge  base. 

Additionally,  summaries  given  by  the  observers  in  the  analytic  Q&A  are  helpful  to  show  differ¬ 
ences  in  the  engagements  from  day  to  day  throughout  the  experiment.  For  example,  if  commu¬ 
nications  were  really  detrimental  during  the  first  two  days  of  the  FBE,  then  improved  on  subse¬ 
quent  days,  then  this  will  be  noted  in  the  overall  picture  and  reflected  in  the  coding.  With  less 
than  comprehensive  and  integrated  qualitative/quantitative  analysis  such  facts  may  not  have  been 
as  easily  extrapolated  from  the  KM  system.  Such  processes  will  enhance  the  understanding  of 
the  FBE  as  a  whole. 

In  the  collective,  the  addition  of  comprehensive,  multi-perspective  data  will  be  a  key  component 
enabling  IJWA  to  completely  utilize  the  “drill-down”  capabilities  possible  in  EQKMS.  For 
example,  the  Data  Sheet  information  differs  from  the  Golf  IRC  data  since  it  is  observational  in 
nature  rather  than  a  log  of  realtime,  dynamic  discussions.  Items  coded  would  include  the  flow  of 
information  through  the  various  systems,  the  dynamics  of  the  joint  interactions  involved  in  the 
FBE,  the  use  of  sensors,  and  the  processes  and  means  of  TCSF  resolution.  This  will  enable  IJWA 
to  resolve  several  issues,  including  the  tracing  of  targets  to  their  original  sensor  event,  while 
providing  needed  insight  on  process  issues  that  hinder  the  efficiency  of  a  given  FBE  system. 

The  above  categories  of  events,  as  noted  by  the  observers,  are  representative  of  the  types  of  data 
that  can  be  extracted  from  EQKMS  for  a  comprehensive  analysis  of  an  experiment.  There  are 
elements  of  operational  matters  in  an  engagement  included  in  the  observers  analysis,  notes 
regarding  the  effectiveness  of  the  techniques  used,  target  specific  observations,  sensor  specific 
observations,  system  specific  observations,  system  operator  notes,  communications  systems 
activities,  and  general  process  and  procedure  observations. 

At  a  more  practical  level,  the  tracing  of  target  numbers  to  sensor  events  is  one  of  the  more 
challenging  tasks  faced  by  the  IJWA  EQKMS  team.  The  primary  means  to  match  a  target 
number  to  a  sensor  event  is  for  the  analyst  to  look  back  and  forth  between  the  Engagement  chat 
and  the  Targets-Sensors  chat  and  correlate  these  with  the  LAWS  data.  With  this  technique,  the 
analyst  uses  time  as  a  reference  point  to  recognize  sensor  events  in  Targets-Sensors  that  match  the 
target  number  being  discussed  in  Engagement.  The  target  specific  observation  data  in  the  Data 
Sheets  information  is  another  means  to  link  target  numbers  to  their  original  point  of  detection 
by  sensors  to  determine  the  overall  target  identification  and  resolution  sequence.  Thus,  an 
additional  verification  mechanism. 

Target  numbers  and  their  corresponding  sensor  events  can  be  filtered  through  various  coding  and 
searching  techniques.  When  a  sensor  event  is  noted  in  the  Data  Sheet  files  it  can  be  tagged  with 
the  target  number.  Then,  when  IJWA  analysts  conduct  searches  on  specific  target  numbers  not 
only  will  the  data  segments  that  have  the  target  number  be  filtered  but  so  will  the  pre-target 
number  sensor  events  that  have  been  tagged  with  the  target  number. 
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IJWA  analysis  can  thereby  address  recurring  problems  that  impact  the  efficiency  of  the  Battle 
Group.  Qualitative  analysis  is  important  because  problems  are  not  always  discussed  in  the 
context  of  an  actual  problem  but  rather  are  inferred  by  the  type  of  discussion  going  on  in  relation 
to  the  situation.  Examples  of  such  situations  include:  misunderstandings;  misinterpretations  of 
data;  system  or  communications  malfunctions;  sensor  allocation  problems;  missile  guidance 
problems;  and  target  mensuration  problems.  We  have  defined  these  types  of  problems  as  "pro¬ 
cesses"  and  there  are  ten  different  processes  that  have  been  defined  within  the  IJWA  EQKMS. 
These  are  discussed  later  in  this  section. 

5.9  LAWS 

The  LAWS  qualitative  resources  are  concerned  with  the  usage  and  effectiveness  of  the  LAWS 
system  in  the  FBE.  The  information  in  this  resource  serves  as  a  cross-search  key  for  information 
in  the  knowledge  repository.  The  data  is  pertinent  for  searches  that  deal  with  process  issues  and 
to  fill  some  of  the  "information  gaps"  in  the  tracing  of  target  resolution  times.  For  example, 
searching  for  specific  target  resolution  times  within  IP-Chat  can  be  difficult  due  to  many  differ¬ 
ent  problems.  A  cross-key  search  to  LAWS  data  can  often  resolve  the  matter. 

An  example  of  a  problem  in  FBE-G  was  when  the  LAWS  systems  have  multiple  mission  num¬ 
bers  for  one  target.  This  fact  can  be  difficult  for  the  analyst  to  determine  just  from  the  data 
available  in  IP-Chat.  The  addition  of  the  LAWS  information  to  the  knowledge  base  provides  a 
search  resource  wherein  mission  number  processes  can  be  noted,  coded  and  correlated  to  specific 
events.  Thus,  the  analysts  can  use  the  LAWS  data  collectors  information  to  verify  that  multiple 
mission  numbers  for  single  targets  was  an  issue  on  the  day  in  question  and  can  integrate  this 
knowledge  into  their  analysis. 

Another  problem  that  surfaces  often  within  the  FBE  is  block  colors  not  changing  appropriately. 
There  were  many  data  segments  in  IP-Chat  that  show  problems  at  the  LAWS  stations  for  the 
operators  concerning  the  command  block  and  the  mission  nominator  blocks  colors  being  wrong 
and/or  changing  frequently.  This  kind  of  problem  is  noted  by  the  LAWS  data  collectors  and  a 
data  collector  may  also  offer  suggestions  for  fixing  the  problem  or  explanations  for  the  cause  of 
the  problem.  This  kind  of  data  would  not  be  available  to  analysts  without  the  addition  of  these 
files  to  the  EQKMS,  and  the  relevant  issues  could  not  be  as  fully  addressed  in  the  final  analysis. 

For  example,  the  file  titled  "IKE  data  logs4040506.doc"  is  a  summary  of  LAWS  issues  noted  by 
the  LAWS  data  collector  for  the  dates  of  April  5th,  6th,  and  7th.  Below  are  samples  of  data 
segments  from  this  file  that  would  prove  beneficial  to  high-level  search  topics  involved  with  the 
use  of  the  LAWS  system,  as  well  as  lower-level  searches  on  LAWS  data  for  specific  days  in  the 
FBE. 


Confusion  at  the  beginning  of  the  day  as  to  who  had  control  over  firing  units 
because  command  blocks  were  being  turned  green,  however,  this  was  due  to  the  fact 
that  JFMCC  (IKE)  had  receiving  capability  but  no  transmit  capability.  (IKE  data 
logs4040506.doc) 
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Much  confusion  running  missions.  Anzio  fired  TLAM  without  a  route  and  while 
CMD  block  was  white  and  WRD  block  was  also  white.  CTF  69  fired  on  a  SCUD 
B  with  TGT  block  red  and  CMD  yellow.  No  missions  were  run  entirely  through 
the  system.  All  missions  were  injected  by  MTO.  Not  much  sensor  coordination. 

Chat  sessions  not  utilized  properly.  A  lot  of  discussion  of  engagement  issues  in  the 
TCT_CMD  chat  session.  Voice  was  used  more  than  it  should  have  been  partially 
due  to  the  fact  that  was  the  only  way  IKE  could  communicate  to  the  outside  world. 

(IKE  data  logs4040506.doc) 

The  information  provided  in  the  passages  above  would  not  be  easily  ascertainable  from  a  typical 
information  system  but  can  be  easily  extracted  from  the  EQKMS  through  the  knowledge  hierar¬ 
chies.  This  is  due  to  the  advanced  cross  file,  cross  project,  cross  FBE  search  and  filtration  capa¬ 
bilities  of  EQKMS.  Data  segments,  like  the  examples  above,  will  prove  to  be  an  important  part 
of  the  knowledge  base  for  FBE  analysis  in  relation  to  the  LAWS  system.  The  easy  access  of 
important  data  segments  like  the  above  will  also  be  key  to  high-level  FBE  analysis  and  can  be 
used  to  track  the  usage  and  effectiveness  of  the  Land  Attack  Warfare  System  throughout  FBEs. 
Thus,  LAWS  qualitative  data  gives  the  analysts  a  much  clearer  picture  of  the  events  in  a  particu¬ 
lar  engagement — especially  as  they  pertain  to  LAWS  information  regarding  the  process  of  target 
resolution. 

Another  example  is  the  "IKE  observer  report0329-0407.doc"  file  that  contains  information 
about  the  LAWS  system  such  as:  actions  completed,  PC  configuration  and  LAWS  software 
installation,  installation  of  map  data  (LAWS),  clearing  the  SMTP  outbox  (LAWS),  installing  the 
mIRC  chat,  and  basic  LAWS  training  with  other  shipriders.  As  stated  below,  there  are  specific 
cause  and  effect  type  problems  that  can  be  defined  by  an  EQKMS  cross-reference  to  the  ob¬ 
server/data  collector  information: 

The  LAWS  operators  are  having  a  difficult  time  performing  their  jobs  because  of  so 
much  information  being  passed  over  the  LAWS  terminal  that  strictly  speaking  does 
not  concern  LAWS  operation.  The  exercise  is  using  chat  windows  and  VAT  for 
internet  voice  communications  and  both  of  these  are  located  on  the  LAWS  terminal. 

The  battle  officer  for  LAWS1  (JFMCC)  is  using  the  LAWS  keyboard/mouse  and 
terminals  more  for  monitoring  the  comms  than  for  operating  LAWS.  (IKE  observer 
report0329-0407.doc) 

This  kind  of  information  is  based  on  insight  from  an  experienced  observer  in  the  FBE  and 
contains  data  that  would  be  otherwise  difficult  to  include  in  an  analysis.  The  inclusion  of  these 
files  into  the  EQKMS  will  enable  an  easy  access  and  retrieval  of  the  key  data  and  concepts.  The 
integration  of  this  information  into  the  knowledge  base  will  enable  a  higher  level  of  analysis. 

“JFACC  4  5.doc”  is  a  JFACC  LAWS  observer  log  from  April  5th.  Topics  summarized  in  this  file 
include:  multiple  mission  numbers  for  one  target;  WRD  column  not  being  used;  Unit  ID 
mismatch;  notice  of  target  modification;  Mission  Manager  Nominator  block  is  not  accurate,  and 
concerns  that  the  track  numbers  on  GISRS  do  not  match  up  with  LAWS  track  numbers;  and 
concerns  that  not  all  units  are  entering  the  GCCS  data  according  to  the  CONOPS.  Here  again 
is  another  file  that  will  bring  the  same  benefits  already  discussed  to  the  overall  understanding  of 
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the  FBE  (high  and  low  level)  and  add  to  the  search  results  from  EQKMS.  One  of  the  unique 
features  of  this  resource  is  that  it  highlights  LAWS  actions  listed  by  time.  The  fact  that  events  are 
time  quantified  means  they  will  be  easier  to  correlate  to  FBE  engagement  operations  for  the 
specific  day  in  question. 

5.10  Sensor  Management 

The  "Sensor  Management"  resource  contains  files  that  have  data  concerning  lessons  learned, 
recommendations,  and  observations  done  during  the  course  of  the  FBE.  Added  to  the  EQKMS, 
this  data  resource  supports  those  analysts  looking  at  sensor  processes  as  well  as  engagement  issues 
or  high-level  cross-FBE  searching. 

For  example,  the  data  in  the  file  "Observations  for  GISRC  -  LAS. doc"  is  inclusive  of  the  observ¬ 
ers  hardware/software  recommendations  for  improving  the  GISRC  system,  suggestions  for  better 
sensor  usage  and  allocation,  discussion  of  communications  usage  between  sensor  operators, 

DTFs,  situational  awareness  (COP),  IRC  usage  and  GISRC/LAWS  map  features.  This  data  will 
be  coded  as  recommendations  and  observation  within  the  EQKMS.  This  coding  scheme  will 
allow  for  cross-file  searching  in  order  to  bring  together  all  of  the  many  data  segments  that  in¬ 
clude  recommendations.  These  types  of  recommendations  need  to  be  coded,  categorized,  and 
warehoused  for  further  detailed  analysis.  Doing  this  in  the  EQKMS  allows  for  easy  availability 
and  retrieval  of  this  newly  categorized  information. 

Another  example  is  the  file  "FBE-G  GISRC-QL-Final.doc"  which  is  an  executive  summary  of 
lessons  learned  through  observation  analysis.  Main  sections  in  this  document  include:  JTF 
Sensor  Planning/Management;  ISR  Anchor  Desk  Screen  Layout;  Offensive  and  defensive  Battle 
Watch  Captains;  decentralized,  network  centric  functional  architecture;  ISR-T  screening  pro¬ 
cesses;  Target  Block  numbering;  Multiple  Target  Keys;  Digital  Target  Folders;  UAV,  ELINT,  & 
MTI;  LAWS  mission  nominations;  Distributed  mensuration  distribution,  Tactical  UAV  control; 
and  digital  video  telemetry.  There  are  detailed  summaries  including  recommendations  for  each  of 
these  topics. 

With  the  integration  of  the  above  file  into  the  EQKMS  the  analysts  will  be  able  to  search  obser¬ 
vation  summaries  that  are  related  to  nearly  every  process  involved  in  the  FBE.  This  data  will  be 
especially  helpful  for  the  sensor  issues  that  are  being  researched.  This  data  can  be  used  for  cross¬ 
search  techniques  when  researching  other  topics  within  the  FBE. 

5.11  Process  Descriptors 

When  the  EQKMS  project  was  first  undertaken  the  focus  was  to  trace  target  numbers  across  the 
knowledge  repository  and  extract  data  relevant  to  the  identification  and  resolution  of  targets. 
However,  once  the  IJWA  analysts  started  reading  and  coding  the  data  it  became  apparent  that 
there  were  problems  that  had  a  significant  effect  on  the  FBE.  These  problems  were  occurring  too 
often  to  be  ignored  and  were  relevant  to  the  overall  effectiveness  of  the  FBE.  So,  these  problems 
were  defined  as  "processes"  and  a  coding  scheme  was  created  to  deal  with  these  processes  and 
defined  within  the  EQKMS  codebook.  The  inclusion  of  “process”  tagging  within  EQKMS  was 
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initiated  by  the  knowledge  workers  reviewing 
the  qualitative  information  and  the  coding  is 
consistent  with  the  high-level  analysis  issues 
established  in  Figure  2. 

As  a  result  of  this  inclusion,  IJWA  analysts  can 
now  search  different  types  of  processes  and 
filter  by  the  frequency  of  occurrence  of  a 
specific  process.  “Processes”  generally  revolve 
about  issues  that  affect  the  dynamics  and 
effectiveness  of  an  FBE  system.  Examples  of 
problems  include:  misunderstandings;  misin¬ 
terpretations  of  data;  system  or  communica¬ 
tions  malfunctions;  sensor  allocation  problems;  missile  guidance  problems;  and  target  mensura¬ 
tion  problems.  The  identified  processes  are  as  follows: 


SEARCH:  PROCESS -1 

SEARCH  CODES: 

# -  PROBLEM  #-TRACK_PROB  T#GJ0015  #-PROCESS-l 

FILE:  #COP0407 

[09:52]  <COPCI>  so,  can  we  find  out  who 
entered  gj0015?? 

[09:53]  <COPCI-trk>  right... but  it  was  a 

trk. .. should' ve  been  a  unit... there 
was  also  another  one  already  done 
correctly 

[09:54]  <COPCI>  okay,  pis  try  to  find 

out  who  input  gj0015...i'd  like  to 
highlight  that  later  today  as  a 
mistake . . . 


Figure  21 .  Partial  IRC  discourse  tagged  as  Process-1 . 


Process-1  :  Target  identification  or  launch  sequence  problems,  concerning  matters  such  as: 
(a)  who  should  take  the  target,  (b)  interpretation  of  rules  of  engagement,  (c)  conflicting 
commands,  (d)  trouble  receiving  target  mensuration  data,  (e)  target  folder  establishment  or 
transmission,  (f)  box  identification  or  transmission,  (g)  insufficient  firing  data,  (h)  incorrect 
color  code  of  target.  See  Figure  21. 

Process-2  :  The  process  of  upgrading  or 
troubleshooting  software,  hardware,  systems 
or  sensory  equipment.  Or,  walking  opera¬ 
tors  through  a  troubleshooting  session.  See 
Figure  22. 

Process-3  :  A  loss  of  time  or  time  ineffi¬ 
ciency  due  to  a  deviation  in  a  computer, 
communication  or  sensor  system. 

Process-4  :  Personnel  misunderstanding  or 
a  misunderstanding  concerning  the  inter¬ 
pretation  of  incoming  data  from  one  or  more  systems,  or  the  misreading  of  data  on  a  system, 
or  problems  with  the  input  of  data,  or  inconsistencies  with  the  source  of  the  data  (e.g.,  target 
data).  For  example,  in  the  following  passage  there  is  some  confusion  as  to  which  system  is 
designating  a  target: 

[12:49]  <COPCI-trk>  anything  beginning  with  GG,  GJ,  GA,  LS,  LA  is  from  laws 
[12:50]  <COPCI>  no,  there  from  gisrs,  and  there  is  no  mensuration  as  yet.... 

[12:54]  <COPCI-trk>  if  it  is  in  GCCS,  with  those  digraphs,  it  came  from  LAWS 
[13:07]  <COPCI>  that's  not  what  i  was  told 


SEARCH:  PROCESS -2 
FILE:  #FB0401 

[16:27]  <Burian>  GIRSC5 :  WHAT'S  YOUR 

ESTIMATE  OF  BEING  OPERATIONAL? 
[16:27]  <ANZ_gisrc9>  this  may  take  a 
couple  paragraphss. 

[16:29]  <GISRC5>  Burian  -  I  need  to 

download  ERM  software  yet.  anzio 
is  helping  right  now.  would  love  to 
be  done  this  evening.  Still  have  a 
few  TASID  install  check  to  make 
[16:29]  <ANZ_gisrc9>  you  need 

GISRC7\d\fbe-g___setup\erm.zip  and 
ERMUpdateforPC.zip  and 
ERMUpdate0326 . zip 


Figure  22.  Partial  IRC  discourse  tagged  as  Process-2. 


Process- 5  :  A  referral  of  an  information  source  (personnel)  to  another  chat  for  further  discus¬ 
sion  based  on  an  identified  topic  and  the  appropriate  continuance  of  that  topic  in  that  chat 
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area.  The  process  issue  under  investigation  would  generally  involve  the  synchronization  and 
consistency  of  the  topic  or  target  referrals  for  continued  action  or  analysis.  An  example 
would  be  as  follows: 

[10:50]  <COPCI-trk>  grnd:  when  you  get  of  interest  elint  (like  what  you've 
been  passing)  make  sure  it  goes  out  of  the  Targets-Sensors  channel. 

[13:40]  <LCDR_Burian>  IKE,  REQ  YOU  COME  UP  IN  #TCT_LEAD 

Process-6  :  A  system  or  human-system  target  identification  issue  such  that  a  target  has  been 
improperly  read  or  numbered  by  a  system  or  operator,  or  there  is  a  deviation  between  num¬ 
bers  and  targets,  or  different  numbers  have  been  assigned  to  the  same  targets. 

[12:35]  <COPCI-trk>  jfacc  -  GJ4037  is  the  same  as  GJ4036  and  4036  is  what 
was  nominated.. ..do  not  enter  4037 

[12:50]  <COPCI-trk>  naval  -  can  you  start  associating  some  of  the  elint  to  the  trks 
[12:59]  <COPCI-trk>  nevermind.. ..they  won't  assoc 

[13:00]  <COPCI-trk>  one  is  real  world,  the  other  is  live-trng...can't  assoc  the  2 
[13:30]  <jfac-gccs>  copci-trk:  I  don't  hold  4037  in  my  system  (I  recall  entering  4036) 

[13:31]  <COPCI-trk>  rgr.... sorry  about  that... .turns  out  it  was  anzio  entering  the  wrong  tgt  number 


Process-7  :  Target  action  with  missing  or  incomplete  data  and  significant  discussion  based  on 
this  issue  or  associated/supporting  data  processes. 

Process-8  :  Command  or  command  structure  procedures  regarding  targeting  and  delivery, 
such  as:  (a)  the  assignment  of  a  target,  (b)  commands  to  remove  target  given  varying  com¬ 
manders  and  launch  operatives,  (c)  confusion  over  whether  or  not  a  target  is  approved  and  a 
green  has  been  given,  (d)  confusion  over 


firing  processes  concerning  target  priority/ 
assignment.  See  Figure  23. 

Process-9  :  Battle  Damage  Assessment 
indicating  that  the  target  destroyed  may  not 
coincide  with  the  target  initially  identified, 
or  that  it  is  a  civilian  target. 

Process-10  :  Problems  concerning  missies 
such  as:  (a)  the  guidance  and  control  of 
munitions  once  in  route,  (b)  problems 
viewing  the  status  of  a  missile,  (c)  confusion 
over  how  many  missies  have  been  fired  at  a 
target. 

In  sum,  the  above  manner  of  qualitative  analy¬ 
sis  requires  skilled  knowledge  workers,  and 
analysts  with  an  understanding  of  the  technical 
and  environmental  variables  in  play  during  an 


SEARCH :  PROCESS  -  8 

FILE:  #EN0407 

[09:48]  <C6F-STRIKE>  JFMCC-I  show  no 

green  cmd  light  on  69  but  they 
fired  anyway. 

[09:49]  <IKE_JFMCC>  rgr  C6f,  thought 

that  was  what  we  heard  last  night. 
Will  enforce 

[09:49]  < JFACC - ISR>  ctf69_laws:  are  you 

saying  sub  has  launched  mis  against 
0051? 

[09:49]  <XVSSN>  c6f -strike,  we  cannot 

mensurate.  only  tgts  we  recieve  is 
via  laws 

[09:49]  <ctf69_laws>  unable  to  launch 

gil570  due  to  rte  conflict,  canco 
with  grn  cmd 

[09:49]  <XVSSN>  and  we  have  assumed  that 
this  is  mensurated  data 

[09:49]  <JFACC-ISR>  correction:  0015 

[09:50]  <ctf 69„laws>  jfacc-  no.  launch 
on  gj0015  not  0051 

[09:50]  < I KE_ J FMC C>  rgr  ctf  69.  I  can 

not  canx  the  grn.  understand  you 
can  not  fire 

[09:50]  < I KE__ JFMC C >  box  probs 


Figure  23.  Partial  IRC  discourse  tagged  as  Process-8. 
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FBE.  The  search  on  processes  is  a  major  attribute  of  EQKMS  and  files  are  being  routinely  coded 
for  process  issues.  A  query  on  a  particular  process  will  reveal  the  transactions  surrounding  the 
occurrence  of  the  item  in  question,  relevant  tags  to  any  associated  targets,  the  operators  involved 
in  the  discourse,  and  the  time  period  in  which  the  exchange  occurred. 


6.0  EQKMS  Processes  and  Functionality 

The  EQKMS  was  designed  for  high  level  analysis  focused  on  effectiveness  issues  in  FBEs.  The 
generated  search  information  supports  IJWA  analysis  of  an  FBE  and  the  comparison  of  attributes 
across  different  FBEs.  An  example  would  be  suggestions  for  software  or  hardware  changes, 
bandwidth  upgrades,  and  C2  changes — which  when  coded  and  addressed  within  EQKMS — can 
provide  cross-area  or  cross-FBE  needs  analysis.  With  the  addition  of  data  from  future  and  past 
FBEs  an  overall  picture  of  systems  effectiveness  can  be  derived. 

EQKMS  is  being  designed  to  facilitate  the  extraction  of  both  high  level  issues  (e.g.,  recurring 
process/problems  throughout  multiple  FBEs)  and  more  specific  items  (e.g.,  FBE  specific,  time 
specific,  target  specific,  etc.)  which  are  classified  as  lower  or  operational  level — such  as  dynamic 
processes  captured  in  realtime.  The  ability  to  “drill-down”  from  decisions  to  event  data,  or  from 
events  to  decisions,  requires  a  flexible  methodology  in  both  the  setup  of  the  system  and  in  the 
data  categorization.  A  systematic  approach  of  data  collection,  management,  filtration  and 
analysis  makes  EQKMS  viable  for  high  and  low  level  analysis.  An  example  of  a  high  level  sys¬ 
tems  classification  in  EQKMS  would  be  data  segments  such  as  the  following: 

A  bandwidth  management  capability  is  necessary  for  TCT  prosecution  with  multiple 
sensors  in  a  Strike  Cell  architecture,  particularly  if  migrated  from  a  shore  reachback 
environment.  With  close  monitoring  with  the  Packeteer  tool,  Strike  Cell  came  close 
to  a  bandwidth  limit,  where  they  might  have  to  decide  between  further  aimpoint 
mensuration  or  additional  imagery  data,  but  did  not  need  to  dynamically  reallocate 
any  bandwidth.  If  the  architecture  moved  to  multiple  radars  to  provide  overlapping 
coverage  in  baffle  areas  during  aircraft  turns,  or  to  provide  independent  views  of 
both  Spot  Hi-Res  SAR  and  SAR/MTI,  a  recommended  capability,  then  bandwidth 
would  have  been  much  more  of  an  issue.  This  is  especially  true  if  automated 
filtering  tools  were  not  available.  Also,  if  multiple  UAVs  or  aircraft  were  reporting 
back  to  the  same  Strike  Cell,  bandwidth  for  aimpoint  mensuration  would  have  been 
an  issue.  If  the  Strike  Cell  is  moved  to  a  naval  platform  (CG,  SSGN,  aircraft)  this 
would  also  be  an  issue  to  ensure  this  tool  were  available  to  support  changes  in  the 
dynamics  of  the  TCT  prosecution  timeline  from  a  sensor  intensive  period,  to  an 
aimpoint  generation  focused  operation,  (source:  FBE-G_Conus_Quicklook.doc) 

The  data  segment  above  is  a  suggestion,  by  an  observer,  that  relates  to  the  resolution  of  a  process 
problem  that  has  already  been  identified  and  tagged  within  EQKMS  by  an  IJWA  analyst.  An 
example  of  low-level  analysis  would  be  the  tracing  of  a  target  from  initial  sensor  reading  to 
conclusion,  or  the  identification  of  a  specific  communication  problem,  for  example,  concerning 
time  latency  in  receiving  mensuration  data.  EQKMS  allows  for  the  warehousing  of  all  this 
knowledge  as  well  as  the  search,  retrieval  and  extraction  of  such  information.  It  will  be  possible 
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through  EQKMS  to  extract  not  only  each  occurrence  of  a  discussed  low-level  problem  but  also 
provide  any  available  high-level  analysis  which  supports  the  issue  under  assessment. 

The  integration  of  high  and  low  level  information,  and  qualitative  and  quantitative  data,  brings  a 
higher  level  of  understanding  to  FBE  operational  issues.  When  such  analysis  is  coupled  with 
data  from  other  FBEs,  the  collective  enables  the  analyst  to  see  if  a  process  or  problem  is  occurring 
across  FBEs — and  to  see  if  any  suggestions  have  been  made  to  fix  the  problem(s).  In  a  situation 
in  which  suggestions  are  being  made  but  not  enacted  (i.e.,  these  problems  keep  recurring 
throughout  various  FBEs)  then  an  EQKMS  search  of  the  repository  can  help  support  the  solu¬ 
tion. 

Another  example  of  the  benefit  of  data  repository  analysis  through  EQKMS  is  the  potential  it 
offers  to  bridge  the  gap  between  software  designers  and  users.  It  is  possible  that  the  needs  of  the 
military  operators  are  not  being  met  by  the  designers  of  the  software  being  used  in  an  FBE.  In 
other  words,  when  suggestions  have  been  made  concerning  upgrades  to  an  interface  or  the 
improvement  of  operational  components  of  software  critical  to  TCT  operations  then  an  EQKMS 
search  of  the  available  resources  can  document  systems  or  software  issues.  An  example  may  be 
found  in  the  following  passage,  which  focuses  on  the  need  for  more  automated  data  insertion  to 
improve  the  COPs  effectiveness: 

TCT  prosecution  requires  more  automated  insertion  of  data  and  fusion  of  data  into 
information  that  can  be  displayed  on  a  COP.  The  current  processes  and  tools  are 
too  MMI  intensive.  High  resolution  SAR,  once  downlinked  from  the  aircraft,  either 
directly  to  a  reachback  strike  cell,  or  through  a  battleforce  network  node  (e.g.  DD, 

CG),  must  be  available  to  provide  situational  awareness  or  target  validation  on  a 
faster  timeline.  The  SAR  is  most  valuable  in  prosecuting  TCTs  if  it  is  directly 
injected  into  the  COP  (if  relevant),  or  rejected  by  an  automatic  FOTC-tool  that 
could  eliminate  images  that  do  not  meet  certain  relevancy  criteria.  MTI  in  isolation 
was  not  useful  without  additional  cueing.  (There  appear  to  be  an  analogy  with 
Active  SONAR  in  the  ASW  methodology  that  we  had  not  considered,  such  that 
MTI  is  best  used  after  initial  classification  of  target  if  interest.) 

Range  of  system  operation  made  it  difficult  to  detect  moving  targets,  and  specifically 
convoy  formations,  in  part  also  due  to  shadowing  from  treelines.  The  image  ma¬ 
nipulation  process  was  too  cumbersome  and  manually  intensive.  To  some  extent, 
there  were  too  many  Strike  Cell  workstations  working  the  problem,  such  that  the 
decision-maker  would  settle  on  an  operator/display  combination  that  provided  a 
comfort  level,  as  opposed  to  a  managed  or  self-synchronized  organization.  Use  of 
chat  was  cumbersome  when  used  for  data  manipulation,  such  as  passing  Lat/Long 
data  for  cross-cueing  of  sensors.  This  COP  also  needs  to  be  a  dedicated  command 
display  that  is  shared  by  all  manned  stations  on  the  aircraft  as  well  as  at  Strike  Cell. 

This  display  capability  was  supposed  to  be  outfitted  on  the  aircraft,  software  applica¬ 
tions  for  common  displays  did  not  make  it  for  integration.  At  Strike  Cell,  one 
monitor  was  hooked  up  to  a  large  vertical  display  for  the  watchsection  and  a 
Commanders  Display.  It  is  important  that  the  GCCS-COP  manager/operator 
function  be  a  separate  display,  so  that  menu  operations  do  not  obscure  the  CDRs 
view  with  dialogue  windows. 
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A  capability'  that  includes  SIPRNET  to  the  aircraft  for  a  COP  and  IRC  chat  would 
be  helpful.  A  video  link  of  aircraft  displays  was  available  over  the  TCDL  at  the  Ship 
Ground  Station,  but  not  directly  to  the  Strike  Cell,  yet  this  capability  we  believe 
would  have  assisted  Strike  Cell  greatly.  The  common  displays  function,  that  pro¬ 
vides  each  aircraft  station  with  a  dual  monitor  and  allows  selection  of  a  COP  at  each 
station  would  have  reduced  verbal  and  written  communications,  and  probably 
reduced  turnaround  time  for  SAR  requests  significantly.  This  would  also  overcome 
the  problem  with  different  systems  using  different  data  formats  for  position  informa¬ 
tion,  which  was  unwieldy  when  trying  to  cross-correlate.  There  was  strong  desire  for 
a  “whiteboard"  collaboration  capability,  that  would  allow  the  decision-maker,  such  as 
a  Strike  Cell  ISR  Action  Team,  to  designate  on  a  screen  collection  requests  for 
upcoming  aircraft  passes  (a  la  John  Madden).  A  GBS  capability  should  be  explored 
as  part  of  the  architecture."  (FBE-G_Conus_Quicklook.doc) 

Analysis  of  the  passage  above  should  lead  to  solutions  that  will  improve  the  overall  operations  of 
the  FBE.  But,  if  such  analysis  is  not  being  used  to  correct  the  issues,  then  this  data  should  be 
made  available  to  persons  who  will  authorize  the  necessary  changes — but  also  communicated  to 
the  software  engineers  and  operational  personnel  that  will  implement  these  changes.  In  reality, 
the  process  is  often  more  complicated  as  the  communication  of  such  needs  is  too  often  obscured 
by  the  difference  in  understanding  between  the  programmers  of  the  software  and  the  users/ 
operators  of  the  software.  EQKMS  analysis  can  help  by  providing  a  bridge  between  the  qualita¬ 
tive  and  quantitative,  and  a  resource  to  categorize  and  classify  situational  and  process  problems. 
If  data  analysis — similar  to  the  above  passage — is  available  upon  request  it  can  be  used  to  help 
bridge  some  of  the  communication  gaps  between 
operators  and  software  developers,  and  thereby  help 
resolve  the  problems  that  need  to  be  addressed. 

6.1  Phase  1  interface  and  Operations 

EQKMS  is  still  in  the  very  early  stages  of  develop¬ 
ment.  The  midterm  objective  is  to  provide  a  means 
to  classify  and  categorize  qualitative  information, 
and  to  then  link  this  information  to  the  quantitative 
data.  The  knowledge  hierarchies  discussed  earlier  are 
a  mean  to  structure  the  data, 
and  the  code  words  [as  identi¬ 
fied  earlier]  are  a  means  to 

provide  drill-up  and  drill-down  capabilities.  Both  the  code  words  and  the 
knowledge  hierarchies  can  be  seen  as  they  are  applied  through  the  code 
trees  in  Figure  24.  The  knowledge  hierarchies  are  the  prominent  means  to 
structure  information  in  the  repository,  and  together  with  the  code  words, 
to  filter  and  search  for  information  in  the  EQKMS  Phase  1  knowledge 
operations. 

The  EQKMS  phase  1  methodology'  utilizes  qualitative  analysis  software  to 
tag  and  code  sentences  and  phrases  in  chat,  interview,  observation  and 


Select  Search  Options 


•  Type  vt  &*wch 

S* ogteCwK  jwfr 
:  WMttotrOW* 


C*ft*f*»  Qvtpvtte 


■  r-itom 

j  »*o 

;  «0*M 


'■*/  |  iT:.  f 

Figure  25.  EQKMS 
search  and  filtering 
options. 


iPr*  ■>■•••  ■  -  1 

Cotf*  Wck*s  Select  Gwfcr  from  the  Tree ; 

| Sort,;  Wore  :  : 

ttwna**  sett 

'  ■  .V,  A\.  i 

\Trni l:|  j  ^ 

viatfok m  ; 

•tnurju.ittaA* 


but 


j£J 


'  V  OR 


1 


k*lOCAYK)N 

TRACKS 

-  TARCfT  8DA 

-  TARGET  «OSS 

-  TARGET  Kit. 

U-  TARGCTtOC 
r-  targfTncm 

h  -•  TARGtT  NUS t 
TARGRT 

r  -  TAKG  Of  OP 
taskT 


Figure  24.  EQKMS  implementation  of  coding 
structure  to  FBE-G  information  and  data  resources. 


41 


reference  documents.  The  knowledge  managers  structure  relationships  in  the  files  and  then 
perform  single,  multiple  or  filtered  searches  based  on  the  code  words  and/or  the  identifiers,  and/ 
or  the  search  filters  (Figure  25).  The  identifiers  and  the  filters  utilize  the  key  category  labels 
specified  in  the  knowledge  hierarchies.  The  search  trees  and  code  word  identification  structures 
are  generally  hierarchical,  but  the  searches  pull  from  across  files — somewhat  in  the  style  of  a 
relational  query.  The  searches  reference  the  subject  data  and  produce  a  thread  which  links  the 
search  result  back  to  the  original  file  for  additional  analysis.  This  capability  is  also  the  basis  of 
the  drill-down  and  drill-up  functionality  from  decisions  to  events  and  vice-versa.  Complex 
searches  for  multidimensional  relationships  is  supported  in  EQKMS.  This  would  include: 

a)  cross-file  searches,  for  example,  from  sensor  to  target  to  decision  to  engagement 

b)  searches  for  variables  in  decision  processes  keyed  to  time  and  operator  actions 

c)  delineation  of  system  and  technical  variables  in  the  TCSF  identification  and  engagement  stream 

d)  multi-tier  searches  for  specific  targets  and  associated  processes 

e)  correlation  by  the  number  of  occurrences  of  a  particular  target,  track,  or  sensor  reading 

f)  any  combination  of  time  variables  in  target  identification,  engagement,  or  assessment 

g)  linkages  to  target  occurrences  across  the  repository,  keyed  to  time  or  other  variables 

6.2  Quantitative  Analysis  of  Qualitative  Data 

Various  systems  are  available  to  manage  qualitative  and  quantitative  data.  However,  these  sys¬ 
tems  have  historically  operated  independently — with  traditional  quantitative  data  processed 
through  transaction  systems,  and  the  supporting  qualitative  information  available  only  through 
an  examination  of  the  notes,  memoranda,  letters,  videoconferences,  observations  and  reports 
existing  on  various  desktop  machines  and  servers  throughout  an  organization.  Quantitative 
information  has  a  history  of  consolidation,  while  methods  to  track  and  utilize  the  various  forms 
of  qualitative  data  are  only  beginning  to  emerge.  The  linking  of  the  qualitative  with  the  quanti¬ 
tative  is  an  even  newer  endeavor  and  the  tools  and  approaches  are  still  relatively  developmental. 

In  the  military,  the  knowledge  management  process  is  several  levels  of  magnitude  more  complex 
than  the  commercial  sector — for  the  reasons  mentioned  above,  but  also  due  to  the  amount  of 
quantitative  and  qualitative  information  which  is  being  generated.  A  corporation  typically  has  a 
central  transaction  processing  repository  and  a  data  warehouse.  The  military  has  dozens  of 
transaction  systems  in  operation  during  an  engagement  and  these  specialized  machines  generate 
quantitative  data  in  various  formats — and  this  variety  does  not  permit  an  easy  integration. 

The  qualitative  processes  are  similarly  complex,  with  various  personnel  providing  varying  levels 
of  perspective  on  each  of  the  quantitative  systems,  and  on  the  interplay  between  the  systems  and 
their  human  operators.  Also  to  be  addressed  are  the  effects  of  varying  interpretations  and  actions 
based  on  the  data,  results  of  human-human  interchanges  concerning  the  data,  and  the  impact  of 
new  variables  in  dynamic  situations.  Corporate  decisions,  by  comparison,  occur  in  a  relative 
static  environment  and  the  analysis  necessary  for  strategy  refinement  has  the  luxury  of  an  ex¬ 
tended  time  frame.  However,  in  the  military,  the  competitive  forces  to  be  addressed  may  be 
changing  every  few  moments,  and  the  strategies  in  play  far  more  complex  and  dynamic. 

The  integration  of  qualitative  and  quantitative  (Q&Q)  analysis  provides  both  the  data  used  to 
make  decisions  and  rationale  and  understanding  of  the  decision  processes.  The  quantitative  data 
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is  the  information  that  flows  through  the  various  electronic  hardware  systems.  This  data  is  in 
support  of  the  C2  timelines.  The  primary  systems  include: 

■LAWS  •  PTW  -COP  -STOW 

•GISRS  ♦  GCCS  •  JSCE 


These  systems  provide  the  core  quantitative  data  for  analysis.  The  supporting  qualitative  data 
would  contain  human-in-the-loop  actions  within  each  operation  and  across  operations — includ¬ 
ing  decision-making  processes,  point-in-time  assessment  of  the  systems,  etc.  In  addition,  within 
each  of  these  systems  are  processes  that  are  part  of  decision  making.  For  example,  mensuration 
often  requires  the  use  of  reference  imagery.  An  operator  access  of  the  reference  image  library 
becomes  an  event  that  can  be  recorded — not  only  the  time  that  particular  type  of  event  occurred, 
but  also  what  type  of  data  was  pulled.  Qualitative  analysis  would  reveal  how  that  data  was  used 
and  the  final  outcome  of  the  application. 


Synthetic  variables  can  be  introduced  to  help  ascertain  likely  interactions  between  qualitative 
decisions  and  quantitative  data.  For  example,  information  injects  from  STOW  can  initiate  trains 
of  events  which  would  generate  understandings  from  the  quantitative  data  and  the  qualitative 
decision-making  processes.  An  understanding  of  the  linkage  of  qualitative  and  quantitative  data 
is  at  the  core  of  the  IJWA  KM  efforts.  The  EQKMS  Phase  1  capacity  to  establish  the  relation¬ 
ships  between  the  type  of  data  and  the  collection  and  assessment  mechanisms  is  illustrated  in 
Figure  26. 


Quantitative 

Qualitative 

Computer-based  data 
generation/capture 

Information  flows  within  and 
between  electronic  systems 

Analysis  of  human-in-the-loop  actions  with 
electronic  data 

External  and  time-critical 
events 

Sensor  readings,  mensuration 
data,  targeting  information 

Expert  observer  logging  data 

Expert  analysis  of  electronic  events 

Expert  guidance  in  decision  processes 

System  and  process 
performance 

Post-experiment  interviews  with  operators 

Expert  observation  of  human-machine  interaction 
Web-site  comments 

Analysis  of  recorded  corns  from  the  operation 
Post-experiment  workshops 

Analysis  of  event  data 

Figure  26.  Qualitative  and  quantitative  resources. 


Event,  system  and  process  performance  information  can  be  captured  and  made  available  through 
knowledge  management  processes  that  integrate  qualitative  information  with  quantitative  data. 
A  crucial  part  of  this  process  may  involve  quality  assurance  procedures — such  as  the  analysis  of 
LAWS  data  to  reveal  misconceptions  about  the  performance  ofTCT  processes.  KM  efforts  in 
IJWA  have  focused  on  TCT  processes  in  the  initial  efforts. 

A  first  stage  in  the  integration  of  quantitative  and  qualitative  data  has  involved  the  analysis  and 
synthesis  of  targeting  events.  Constructing  TCT  timelines  is  a  difficult  process  because  much  of 
the  needed  electronic  data  has  not  been  captured.  In  the  present  system(s)  the  following  data 
sources  are  being  synthesized  in  order  to  recreate  TCT  timelines  and  associated  events  and 
decision  processes: 
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•  Recorded  LAWS  data 

•  Recorded  IRC  Chat  sessions 

•  Observer  data  logs 

•  Data  sheets 

•  Post-experiment  interviews 

•  Reference/technical  documentation 

A  primary  issue  is  that  the  LAWS  data  is  incomplete  and  the  IP  Chat  sessions  are  commonly 
offset  in  time  because  of  the  mechanics  of  participating  in  chat.  Other  issues  are  that  the  ob¬ 
server  logs  record  data  at  only  a  few  of  the  many  locations.  However,  when  the  collective  of  this 
data  and  information  is  integrated  into  the  KM  system  an  accurate  data  stream  or  observation 
can  potentially  be  used  to  correct  chat  time  offsets  or  missing  observer  remarks.  Intensive 
comparison  of  the  data  and  information  sources  can  enable  the  reconstruction  of  a  few  of  the 
TCT  timelines.  This  is  a  primary  benefit  of  the  initial  IJWA  knowledge  management  efforts  to 
integrate  qualitative  and  quantitative  data. 

It  is  important  to  note  the  crucial  role  that  synthesis  by  the  human  analyst  plays  in  this  process. 
The  first  cut  of  synthesis  results  is  to  take  the  system  and  process  performance  information  noted 
above,  place  it  in  categories  (the  beginning  of  synthesis),  and  extract  information  in  which  there 
is  some  confidence.  This  is  the  basis  of  knowledge  management  that  interfaces  quantitative  data 
with  qualitative  information  about  that  data.  EQKMS  is  a  project  within  IJWA  that  is  actively 
conducting  this  integration. 

In  addition,  EQKMS  qualitative  search  results  can  be  measured  quantitatively.  Search  results  on 
a  specific  target,  system,  process,  decision,  or  communication  issue  (for  example)  can  be  counted 
to  provide  a  hard  measure  of  the  number  of  occurrences  of  a  given  event,  a  specific  situation,  or  a 
result.  Complex  or  multidimensional  queries  can  be  similarly  quantified  to  provide  an  objective 
assessment  of  the  scope  of  the  matter  under  consideration. 

Figure  27  illustrates  a  native  capability  of  EQKMS  for  conducting  quantitative  analysis  of 
qualitative  data.  Specifically,  there  are  internal  mechanisms  which  register  the  number  of  occur¬ 
rences  of  a  specific  search  criteria,  the  prevalence  or  length  of  relevant  passages  of  a  given  occur¬ 
rence  within  the  repository,  and  occurrences  of  events  or  decisions  by  file  number  and  location  to 
support  drill-down  and  drill-up  assessments.  The  system  also  supports  traditional  quantitative 
measures,  such  as  the  number  of  hits  and  frequency  of  hits. 

6.3  Phase  2  and  Future  Development 

The  expansion  of  the  knowledge  base,  linkages  to  additional  qualitative  and  quantitative  data 
sources,  and  more  sophisticated  search  and  retrieval  capabilities  using  artificial  intelligence  are  all 
EQKMS  enhancements  in  current  development.  Mechanisms  to  automate  the  processing 
operations  are  also  in  development.  Figure  28  depicts  the  current  evolutionary  path  in  which  the 
core  EQKMS  model  links  qualitative  information  from  various  sources  and  applies  organization 
methodologies  and  coding  structures  to  the  data  to  support  information  extraction.  The  catego¬ 
rization  schema  and  processing  methodology  will  support  information  in  a  variety  of  formats, 
and  potentially,  in  various  locations. 
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In  development  are  methods  to 
more  tightly  integrate  the  output 
from  quantitative  systems  with  the 
supporting  qualitative  information 
which  provides  perspective,  ratio¬ 
nale  and  justification  for  the  event 
data.  This  is  illustrated  in  the  figure 
as  a  linkage  into  relational  and 
object  databases  through  Java,  SQL 
or  other  search  and  processing 
techniques.  This  may  most  rapidly 
occur  as  linkages  into  reports 
generated  from  the  various  svs- 
terns — and  eventually  as  automated 
processing  between  systems.  In  the 
more  advanced  stages  of  design  are 
methods  to  load  data  into  the  XML 
data  object  model  for  web-based 
reporting,  and  the  use  of  agents  and 
neural  nets  as  a  means  to  automate 
the  processing,  filtering  and  search¬ 
ing  operations. 


Figure  29  illustrates  the  logical  Figure  27.  Complex  search  and  filtering  relationships  and  quantitative 

extension  of  the  system  and  meth-  analysis  of  qualitative  findings. 

odology  to  include  a  manner  of 

virtual  knowledge  and  intelligence.  This  may  occur  as  the  information  summaries,  reports,  and 
analysis  utilize  object-oriented  and  agent  technologies  to  support  distributed  queries  and  infor¬ 
mation  dissemination.  In  this  scenario,  through  the  use  of  advanced  information  processing 
tools,  the  data  and  information  resources  can  be  assembled,  categorized,  searched,  and  the  results 
correlated  with  ever-greater  speed.  Ideally,  this  may  one  day  provide  analysis  that  supports  a  very 
rapid  inclusion  of  all  pertinent  information  resources  and  a  very  rapid  dissemination  of  results. 
For  example,  the  sensor  systems  presently  support  a  very  rapid  transfer  of  information  from  the 

technology  to  the  decision 

r  Existing  processes  i__  makers.  The  potential  exists 
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Figure  28.  Evolution  of  EQKMS  to  incorporate  agents  and  neural  nets. 


makers.  The  potential  exists 
for  qualitative  information  to 
one  day  support  the  operational 
data  to  provide  expert  opinion, 
historical  perspective,  pertinent 
references,  results  from  similar 
situations,  or  actions  and 
responses  from  related  environ¬ 
ments.  Such  capabilities  would 
be  the  logical  extension  of  the 
technologies  and  methodolo- 
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gies  advanced  in  rhe  IJWA  EQKMS  project. 


Through  the  KM  processes  described  above 
the  analysis  process  can  both  expand  to 
include  far  more  data  and  thereby  increased 
efficiency,  while  simultaneously  increasing  the 
speed  with  which  the  analysis  occurs.  Simi¬ 
larly,  the  distribution  of  the  analysis  reaches 
key  decision  makers  within  months  or  even 
weeks — which  is  very  rapid  for  the  scope  and 
extent  of  the  operations.  As  the  EQKMS 
project  expands,  it  would  be  possible  to 
shorten  the  time  period  for  analysis,  while 
also  increasing  the  scope  of  the  distribution  of 
the  analysis.  A  possible  objective  would  be  to  provide  decision-makers  with  expert  analysis  soon 
after  an  FDE,  and  potentially  while  an  FBE  is  in  play  or  even  when  a  specific  operation  is  being 
conducted. 


Figure  29.  Evolution  of  EQKMS  to  provide  dynamic, 
distributed,  virtual  knowledge. 


7.0  Conclusion 

Qualitative  analysis  finds  information  not  available  in  quantitative  data,  including  objective 
perspectives,  first-person  accounts,  decision  processes  and  influencing  variables.  When  inte¬ 
grated  with  quantitative  data  the  qualitative  sources  enable  a  tracking  of  decision  processes — 
from  events  to  decisions  and  back  again.  A  comprehensive  knowledge  resource,  supporting  both 
centralized  and  distributed  information  resources,  will  enable  a  higher  level  of  analysis  than  has 
previously  been  available.  The  IJWA  EQKMS  project  will  supporr  various  searches  and  inquiries 
and  produce  not  only  the  event  but  decisions  resulting  from  the  event  and  the  thread  back  to  the 
source  data. 

A  key  output  of  the  system  will  be  analysis  to  support  improved  decision-making  processes.  An 
example  discussed  herein  was  improved  decision-making  based  on  the  accumulated  experiences 
of  peers  in  similar  situations  or  experiencing  similar  events,  e.g.,  decision  processes  in  fires,  data 
flows  in  critical  systems;  and  measures  of  frequency  of  occurrence  in  selected  environments.  The 
systems  would  support  the  identification  of  possible  problem  scenarios  and  likely  outcomes 
based  on  past  occurrences,  or  projections  based  on  similar  situations.  The  analysis  processes  will 
produce  measurable  relationships  between  electronic  data  and  associated  observations,  e.g., 
human-in-the-loop  operations  at  specific  nodes. 

A  key  facet  of  integrated  qualitative  and  quantitative  analysis  is  multilevel  reporting  which 
provides  a  “drill-down”  from  high-level  conclusions  to  supporting  information;  a  “drill-up”  from 
events  to  situations  or  decisions  resulting  from  those  events;  and  channels  for  synthesis  to  higher 
level  decision-making  criteria  and  processes.  The  integration  of  opinions  and  data  into  the 
analysis  can  produce  statements  and  results  about  the  status  of  an  initiative  or  experiment  out¬ 
comes — as  well  as  quantitative  measures  of  the  qualitative  results. 
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